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ABSTRACT 


It is generally aeeepted that eost growth in federal major defense aequisition programs 
(MDAPs) is partially attributable to inaeeurate cost estimates. Cost analysts exhaustively 
analyze manpower and resources to provide accurate estimates. However, the influence 
of transaction costs is often ignored in traditional cost estimates. This thesis investigates 
the association between cost growth and transaction costs, the real cost of business 
negotiations and program management. We collect MDAP cost threshold breach data 
and cross reference it with a proxy for MDAP transaction costs (Systems 
Engineering/Program Management Costs) to determine whether a correlation exists. We 
use multiple logistic regression models to analyze the binary outcome of breach or no 
breach. The results show that for MDAPs with cost-plus contracts there is a statistically 
significant relation between the likelihood of a cost threshold breach occurring and the 
relative magnitude of the MDAP’s transaction costs; no such relation exists for fixed 
price contracts. Although these results show an association between cost threshold 
breaches and transaction costs, there is no evidence of causality between these two 
variables and our exploration of causality is a topic for future research. 
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EXECUTIVE SUMMARY 


The Department of Defense (DoD) 2012 major defense aequisition program (MDAP) 
portfolio is estimated at $1.6 trillion, and eontains 86 MDAPs. While this is a reduction 
in the total cost and number of MDAPs compared against 2011, the total cost of these 
programs has increased by more than $400B since their first full cost estimate 
(Government Accountability Office 2013, 6). Given the large scale of spending, life cycle 
cost estimates for MDAPs, which include the acquisition cost estimates, should be 
credible information for budgetary and policy decisions. 

DoD decision makers use three separate cost estimates: from the program office, 
from a DoD component, and an independent estimate performed by Office of Secretary 
of Defense Cost Analysis and Performance Evaluation (OSD CAPE). Despite these 
efforts, the number of MDAPs that sustain cost breaches (cost overruns of 10% or 
greater) suggests that acquisition costs are frequently underestimated. 

Transaction cost, the real cost of business negotiations and program management, 
is a component of cost estimates that is often overlooked when estimating costs of a 
program or system. This research explores the possibility that transaction costs are one 
possible cause for low cost estimates of MDAPs and the resulting cost breaches. This 
thesis seeks to answer the following questions: 

Is there a relation between MDAP transaction costs and cost breaches? 

Are cost breaches related to the amount, or percentage of total costs, spent on 
MDAP transaction costs? 

By using a proxy for transaction costs (systems engineering and program 
management [SE/PM] costs), this thesis attempts to measure the relationship between 
transaction costs and cost breaches. While most accounting systems do not capture 
transaction costs as a separate category, the SE/PM category would certainly include such 
costs. It is logical to suppose that programs that are more complex or have technical 
difficulties are more likely to experience higher transaction costs which may not have 
been anticipated in the original cost estimate. Consequently, such programs would be 
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more likely to see eost breaehes. This research analyzes the level of SE/PM expenditures 
in major defense acquisition programs to determine whether a relationship exists between 
transaction costs and cost breaches. A brief discussion about the data collection process 
and three models used to answer the proposed thesis questions follows. 

The research was done in two parts: the collection of MDAP cost information to 
build a database; and the selection of an appropriate model to analyze the data. The 
database paired program breaches and SE/PM costs. Program cost breach information 
found in the MDAP selected acquisition report was coupled with SE/PM cost ratios that 
were calculated using data obtained from MDAP cost summary data reports. Eor 
purposes of this research, a cost breach is defined as a cost overrun of 10% or greater. 

Recognizing that transaction costs may not be the only factor associated with cost 
breaches, data were collected for other factors such as contract type and “maturity”— 
defined in this report as the time in years since program initiation. While there are many 
more variables that may be related to cost breaches, contract type and maturity were 
selected based on the availability of relevant data from the two databases used. Defense 
Acquisition Management Information Retrieval (DAMIR) and Defense Cost and 
Resource Center (DCARC). Ideally, the development of a multivariate model that has 
predictive power for MDAP cost breaches could serve as a useful tool. 

Eogistic regression is designed for use with binary-outcome dependent variable 
models and was therefore the logical choice to examine the occurrence of program 
breaches. Eogistic regression models use maximum likelihood estimators, or logits, 
which are the natural logarithm of the odds ratio for a particular outcome. Eixed effects 
logistic regression and population averaged logistic regression were both considered. 
Eixed effects logistic regression looks within programs for changes in the EAC SE/PM 
cost ratio and is a useful tool for describing the impact of short run changes in the 
MDAP. In the end, the population averaged logit model was selected because it looks 
across programs to measure changes in the SE/PM cost ratio and is most useful in 
measuring long run effects. 
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In the first model, MDAP eost breaehes were eompared against estimate at 
eompletion (EAC) SE/PM cost ratios and analyzed using logit fixed effects regression 
and logit population averaged regression. This analysis tested the hypothesis that the 
probability of an MDAP sustaining a cost breach is related to its EAC SE/PM cost ratio. 
The logit population averaged regression model indicates a positive relationship that is 
significant at the 5% (p = 0.05) level between the likelihood of a cost breach occurring 
and EAC SE/PM cost ratio. The salient information that the logit population averaged 
model can provide includes the average marginal effects and marginal effects values 
which describe the effect of the independent variables on the dependent variable. Eor 
instance, based on across program observations the logit population averaged model 
shows that the average marginal effect of the independent variable on the dependent 
variable is 0.80. This means that, on average, for a 1% increase in the SE/PM cost ratio 
there is a corresponding 0.80% increase in the likelihood that a cost breach will occur. 

In the second model an explanatory variable was added: program maturity, 
considered in this report to be the length of time (in years) since the program entered 
Milestone B. The rationale to consider the effect of program maturity against cost 
breaches is that older programs are expected to have less uncertainty and risk than 
younger programs. This model tested the null hypothesis that there is no relation of cost 
breaches with EAC SE/PM cost ratio and program maturity as defined in this report. The 
model was analyzed using logit fixed effects regression and logit population averaged 
regression. While program maturity alone did not yield statistically significant results, 
there is evidence that the maturity of a program affects the probability of a cost breach 
occurring in the model. The average marginal effect of maturity in the population 
averaged logit model is 0.70 and for EAC SE/PM cost ratio the average marginal effect is 
0.89.1 means that, on average, a one unit (1%) change in the maturity of an MDAP 
equates to a 0.70% change in the likelihood of a cost breach occurring. Similarly, a one 
unit (1%) change in the EAC SE/PM cost ratio will result in a 0.89% change, on average, 
in the likelihood of an MDAP incurring a cost breach. 

1 The statistically significant results that produced a marginal effect on the dependent variable were 
calculated using the natural logarithm of maturity, as explained in Chapter IV. 
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The third model introdueed an additional explanatory variable to the previous 
model: eontraet type. This model eompared MDAP eost breaches to EAC SE/PM cost 
ratios, program maturity and contract type. The rationale for considering contract type is 
that acquisitions professionals generally accept the notion that firm fixed price contracts 
are frequently negotiated between the government and contractors for less risky programs 
and cost-plus contracts are negotiated for projects which have greater uncertainty and are 
perceived to be more risky (Government Accountability Office 2009b). Eogic suggests 
that riskier MDAPs should have cost-plus contracts and experience more cost breaches 
than their less risky counterparts. This last model tests the null hypothesis that there is no 
relation between cost breaches and EAC SE/PM cost ratio, program maturity and MDAP 
contract type. Only cost-plus contracts were observed to have statistically significant 
results, at the 10% level. Eor cost-plus contracts, including the maturity of a program has 
an effect on the probability of a cost breach occurring. According to the population 
averaged model, the average marginal effect of maturity is 0.21 and EAC SE/PM cost 
ratio is 1.00. This means that, on average, for a one unit (1%) change in program maturity 
the likelihood of a cost breach occurring will change by 0.21%. Moreover, for a one unit 
(1%) change in the EAC SE/PM cost ratio the likelihood of a cost breach occurring will 
change by 1.00%, on average. The results of this last model show that for cost-plus 
contracts as the EAC SE/PM cost ratio increases there is a corresponding increase in the 
probability of a cost breach occurring. Although there is correlation between SE/PM cost 
ratio and cost breaches, it cannot be assumed that transaction costs are a causal factor for 
cost breaches. 

In this research, the model which included the explanatory variables maturity and 
cost-plus contract types yielded statistically significant results and had the largest values 
for the average marginal effects of EAC SE/PM cost ratio. These contracts exhibited 
relationships between (1) the EAC SE/PM cost ratio and the likelihood of a cost breach 
occurring; and (2) program maturity and the likelihood of a cost breach occurring. Euture 
work to better understand the relationship between transaction costs and cost estimation 
may require the development of a more complex model. Hopefully, this thesis will 
provide a useful foundation toward building a better model. 
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I. 


INTRODUCTION 


The acquisition process is the means by which the federal government acquires 
supplies or services by contract. For the Department of Defense (DoD), the acquisition 
process is used to provide support and services for military operations. The sequential 
three-step process for acquiring supplies and services is: (1) the Joint Capabilities 
Integration and Development System (JCIDS), which identifies system requirements; (2) 
the Planning, Programming, Budgeting and Execution (PPBE) process, which allocates 
resources; and (3) the acquisition process, which obtains the supplies or services that 
satisfy defense needs (Schwartz 2013, 3). This acquisition process includes the “design, 
engineering, construction, testing, deployment, sustainment, and disposal of weapons or 
related items purchased from a contractor” (Acquisition Central 2013) and is managed by 
the program manager (PM). 

The program manager is “the designated individual with responsibility for and 
authority to accomplish program objectives for development, production, and sustainment to 
meet the user’s operational needs.” The PM reports program cost, schedule, and performance 
information to the milestone decision authority (MDA) (Department of Defense Instruction 
5000.01 2003).2 Weapon system acquisitions fall into three categories: ACAT I, II or III. 
Major defense acquisition programs (MDAPs) ACAT I acquisition programs are estimated to 
have expenditures greater than $365M in research, development, technology and engineering 
(RDT&E) or $2.190B in procurement, including all planned increments, in FY 2000 constant 
dollars; in addition, programs may be designated as ACAT I by the Under Secretary of 
Defense for Acquisition, Technology and Logistics (Defense Acquisition University 2011b). 
ACAT I programs are commonly referred to as major defense acquisition programs 
(MDAPs). Acquisition programs that do not meet the requirements for ACAT I are 
categorized as ACAT II or III programs. This thesis will focus on cost overruns for ACAT I 
programs. 


2 The MDA has overall responsibility for the program and is responsible for reporting program eost, 
sehedule and performanee information to higher authority, ineluding Congress (Department of Defense 
Instmetion 5000.01 2003). 
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Program managers use the systems engineering process, an overarching method 
that transitions “from a stated capability to an operationally effective and suitable 
system,” (Defense Acquisition University 2011b) to effectively manage their programs. 
The systems engineering process is applied throughout the entire life cycle of a program, 
and includes everything from system design and development to system operation, 
disposal and life cycle costing.3 Cost estimate inaccuracies are identified as a leading 
cause of system cost growth (Blanchard and Fabrycky 2010, 567). This report seeks to 
better understand cost overruns by looking at transaction costs, a significant component 
of program management, and their relation to MDAP cost threshold breaches. In 
particular, the research looks at these questions: 

• Is there a correlation between MDAP transaction costs and acquisition 
cost threshold breaches? 

• Do underestimated transaction costs contribute to MDAP cost threshold 
breaches? 

• Are cost threshold breaches related to the amount, or percentage of total 
costs, spent on MDAP transaction costs? 

A. COST OVERRUNS IN MAJOR DEFENSE ACQUISITION PROGRAMS 

In this report, cost overruns are defined using earned value management (EVM) 
terms. The DoD has used the EVM system since 1996 to evaluate contractor management 
systems (Defense Acquisition University 2011b). According to EVM, a cost overrun 
situation occurs whenever the actual cost of the work performed to complete the project 
is greater than the budgeted cost of the work required to complete the project. The extent 
of the cost overrun can be determined by evaluating the difference between the actual 
cost and budgeted cost. The severity of a cost overrun may warrant certain actions to be 
taken by the program manager. EVM and the consequences of sustaining a cost overrun 
will be discussed further in Chapter II. 

The prevalence of cost overruns in major defense acquisition programs suggests 
that there may be a component of cost estimations that is not accurately considered. As 
defined by the Defense Acquisition University (DAU), a cost overrun is “the amount by 

3 For more information on life cycle costing see Benjamin S. Blanchard and Wolter J. Fabrycky, 
Systems Engineering and Analysis 5th Ed. (Boston: Prentice Flail, 2011), 566-631. 
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which a contractor exceeds the estimated eost and/or the final limitation (eeiling) of the 
eontraet” (Defense Aequisition University 2011b). The Government Aeeountability 
Offiee (GAO) reported that the 96 aetive MDAPs in FY 2011 sustained aequisition eost 
growth of $74.4B, oolleetively. Furthermore, about 60% of this eost growth ean be 
attributed to proeurement eosts (exelusive of proeurement quantity ehanges) and researeh 
and development eosts (United States Government Aeeountability Offiee 2012). A 2011 
GAO report stated that “inaeeurate eost estimates are responsible for the strongest 
eorrelation with net eost growth ehanges and are assoeiated with 40 pereent of the 
aeeumulated eost overruns” (GAO 2011, 2).^ A 2007 RAND study on the eost growth in 
DoD weapons systems analyzed total program and proeurement eost growth for 46 
programs. The study determined that the eost growth ratio aeross all programs was 1.46, 
or 46 pereent, more than the eost estimate for Milestone B (program initiation) (Obaid 
Younossi et. al 2007, xvi). 

A reeent study looked at Department of Defense eontraetors and found that they 
earn above-”normal” profits when eompared to non-defense industry peers. It reports that 
“eost growth oeeurs steadily throughout the program lifespan” and goes on to further 
suggest “that younger programs are not performing better than older programs” (Wang 
and San Miguel 2012). This eorroborates the Defense-Industrial Initiatives Group, Center 
for Strategie and International Studies findings in their report entitled Cost and Time 
Overruns for Major Defense Aequisition Programs. 

Perhaps there are aequisition eosts that may not be eompletely refieeted in life 
eyele eost estimations: transaetion eosts. Is it possible that underestimated transaetion 
eosts have been eontributing to eost growth for MDAPs for deeades? Cost estimates that 
better aeeount for transaetion eosts should be more aeeurate (relative to original 
estimates). This thesis addresses transaetion eosts, a eomponent of system eost that is not 
well eonsidered in eost estimates. Is there a eorrelation between transaetion eosts and eost 
estimate overruns in MDAPs? A better understanding of transaetion eosts may help to 


4 Information about improving cost estimating methodology can be found in the GAO Cost Estimating 
and Assessment Guide, GAO-09-3SP, “Best Practices for Developing and Managing Capital Cost 
Programs.” 
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explain the differenee between aetual and estimated eosts in MDAPs, whieh eould lead to 
more realistie eost estimates and better foreeasting of eost breaehes. 

To measure transaetion eosts, this study uses a proxy measure that ineorporates 
systems engineering and program management eosts. Aeeording to MIL-STD-881C, 
system engineering is defined as “the teehnieal and management efforts of direeting and 
eontrolling a totally integrated engineering effort of a system or program” (Department of 
Defense 2011, 221). Program management is defined as “the business and administrative 
planning, organizing, direeting, eoordinating, eontrolling, and approval aetions 
designated to aoeomplish overall program objeetives, whieh are not assoeiated with 
speeifie hardware elements and are not ineluded in systems engineering” (Department of 
Defense 2011, 222). The similarities in the definitions for systems engineering and 
program management ean lead program managers and eontraetors to different 
interpretations for alloeating program eosts. In either ease, whether the eosts are alloeated 
to systems engineering or program management, it is reasonable to assume that a 
eombination of the two will provide an approximate, albeit indireet, measure of the 
transaetion eosts experieneed by a program. 

This study seeks to understand the impaet of transaetion eosts on MDAP eost 
overruns. Speeifically, it looks at signifieant eost overruns, defined as “cost breaches.” A 
cost breach occurs when program costs exceed the approved acquisition program baseline 
(APB) by 10%, or greater. Cost breaches and MDAP APBs are discussed in further detail 
in subsequent sections of this report. For an MDAP that program managers feel may 
incur a cost breach it seems reasonable to assume that the SE/PM costs may increase as 
managers proactively respond to the program manager’s direction. Programs that have 
recently experienced a cost breach may also see increasing SE/PM costs as a reaction to a 
program manager’s decisions and direction. Can cost breaches be related to the amount of 
SE/PM cost, as a percentage of total cost, in a program? 

B. DEPARTMENT OF DEFENSE (DOD) COST ESTIMATION 

The purpose of cost estimation in the DoD is to provide a reasonably good 
prediction of the costs associated with a program through its operational life—that is, to 
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estimate all the major eomponents of life cyele eosts. There are many different ways to 
define life eyele eosts, however it is generally aeeepted that life eyele eosts are the sum of 
the following elements: researeh and development (R&D) eosts, produetion and 
eonstruetion eosts, operation and support eosts, and retirement and disposal eosts 
(Blanehard and Fabryeky 2011). Aequisition eosts, whieh eonsist of the R&D eosts and 
the produetion and eonstruetion eosts, eompose a signifieant portion of a program’s 
estimated life eyele eost. 

For the purpose of this report, eost estimation in aequisitions is eonsidered to be 
“the summation of individual eost elements, using established methods and valid data, to 
estimate the future eosts of a program, based on what is known today” (Government 
Aeeountability Offiee 2009a). In a 1972 GAO report titled “Theory and Praetiee of Cost 
Estimating for Major Aequisitions,” it was noted that DoD eost estimating teehniques 
aeross DoD programs were ineonsistent in their level of detail and that the eost estimating 
teehniques used by the DoD programs varied widely. 

GAO believed these disparities typieally resulted in inaeeurate, low eost 
estimates. In response to these findings and in the same 1972 GAO report, GAO 
published its first version of “Basie Charaeteristies of Credible Cost Estimates,” a 
deseriptive listing of basie eharaeteristies that GAO advises should be found in program 
eost analyses to this day. Although this guidanee has been in existenee for years, GAO 
ease studies provide evidenee that in many programs the guidanee has not been followed. 
Some of the problems noted in that report are: 

• Known eosts had been exeluded without adequate or valid justifieation; 

• Historieal eost data used for eomputing estimates were sometimes invalid, 
unreliable, or unrepresentative; 

• Understanding the proper use of the estimates was hindered beeause the 
estimates were too low; 

• Readily retrievable eost data that eould serve in eomputing eost estimates 
for new weapon systems were generally laeking; and 

• Organized and systematie efforts were not made to gather aetual eost 
information to aehieve eomparability between data eolleeted on various 
weapon systems or to see whether the eost data the eontraetors reported 
were aeeurate and eonsistent. 
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In an effort to help government managers and auditors ensure eredible cost 
estimates, the GAO published the “GAO Cost Estimating and Assessment Guide: Best 
Practices for Developing and Managing Capital Program Costs” in 2009. The majority of 
the guide (Chapters 1-17) is used to discuss best practices for obtaining credible cost 
estimates and the relation between these estimates and earned value management (EVM) 
(Government Accountability Office 2009a, 1). EVM is the system used by DoD to 
evaluate contractor management systems (Defense Acquisition University 2011b) and is 
explained in the next chapter. The latter portion of the guide (Chapters 18-20) provides 
guidance for “integrating cost estimation, system development oversight, and risk 
management” to the program’s life cycle costs (Government Accountability Office 
2009a, 2). 

In 1992, Office of the Secretary of Defense Cost Analysis Improvement Group 
(OSD CAIG) promulgated the “DoD Cost Analysis Guidance and Procedures” (DoD 
5000.4-M) to improve the reliability of DoD cost estimates. The CAIG function is now 
conducted by Director, Cost Assessment and Program Evaluation (D, CAPE) within OSD 
and is responsible for overseeing DoD cost estimation policy and procedures and 
providing independent life cycle cost estimates for MDAPs (Defense Acquisition 
University 2011b, B-54).5 In addition to the independent life cycle cost estimate, DoD 
Instruction 5000.2 and DoD 5000.2-M require that a program office estimate and DoD 
component cost analysis estimate also be performed (Department of Defense 1992, 8). 
These three estimates are based on the MDAP cost analysis requirements description 
(CARD), a document which provides an overview of the “technical and programmatic” 
features of an acquisition program (Defense Acquisition University 2011b, B-56). The 
Director of CAPE considers these estimates and provides this information for 
consultation to the Under Secretary of Defense for Acquisition, Technology and Eogistics 
(USDAT&E) when USDAT&L is the milestone decision authority (MDA) (i.e., ACAT 
ID MDAPs) (Congress 2009, Section 101). In accordance with DoDI 5000.02, Enclosure 
4, Table 2, independent cost estimates for ACAT ID programs must be provided by OSD 

^ A responsibility of the Offiee of Cost Assessment and Program Evaluation (CAPE) is to provide 
“independent analytie adviee” to the Seeretary of Defense on the eost-effeetiveness of defense systems 
(Cost Assessment and Program Evaluation 2013a). 
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CAIG before entering the Teehnology Development phase (Milestone A), the Engineering 
and Manufaeturing Development phase (Milestone B, or program initiation), the Production 
and Deployment phase (Milestone C) and the full-rate production decision review. 

Cost estimates are expected to change and presumably become more accurate as 
the program matures. These refinements are largely due to lessening uncertainty 
surrounding the specific program requirements and less outstanding expenditures 
remaining as the program progresses to completion. In order to ensure a realistic and 
accurate cost estimate it is important that MDAP cost estimates be periodically re¬ 
evaluated and updated to account for requirements changes, actual cost expenditures 
instead of estimated costs and the latest version of the program’s schedule (Government 
Accountability Office 2009a, 37). GAO states that “relying on a standard process that 
emphasizes pinning down the technical scope of the work, communicating the basis on 
which the estimate is built, identifying the quality of the data, determining the level of 
risk, and thoroughly documenting the effort should result in cost estimates that are 
defensible, consistent, and trustworthy” (Government Accountability Office 2009a, 12). 

C. TRANSACTION COSTS 

Transaction costs are the costs associated with “source selection, periodic 
competition and renegotiation, contract negotiation and management, performance 
measuring and monitoring and dispute resolutions” (Angelis et al. 2008). Transaction 
costs are derived from the complexity and riskiness of the work to be accomplished. 
Thus, transaction costs represent a “cost of doing business” that may not be completely 
captured in standard WBS estimates. 

Traditional cost estimation is derived from the Work Breakdown Structure 
(WBS); however, due to the prevalence of cost overruns and inaccurate cost estimates, it 
can be deduced that something is missing from this analysis of cost estimation. 
Transaction costs may be the delta between actual and estimated costs that is difficult to 
capture and assess. 

In general, a program has two types of costs: contract costs and transaction costs. 
Contract costs are the costs of production and are usually captured in the WBS. 
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Transaction costs can be considered motivation or eoordination costs and may not be 
adequately captured in the WBS (Angelis et al. 2008). 

Total Program Cost = Transaetion Costs + Contraet Costs; 

Transaetion Cost = Coordination Costs + Motivation Costs; 

Coordination costs include search and information costs; bargaining and decision 
costs; policing and enforcement eosts. These eosts are affected by market eompetition 
(eontestability), asset specificity, and the reeurrenee or frequeney of a transaetion. Asset 
speeificity, a manufaeturer’s speeialization in the production of a system or maehine, may 
result in the government beeoming reliant on a sole provider and lead to opportunistie 
behavior by the manufaeturer. Motivation eosts are those eosts whieh promote produetive 
efforts and provide ineentives to eneourage investment, as well as those eosts whieh deter 
unproduetive bargaining and opportunistic behavior. Motivation eosts are impaeted by 
the eomplexity of the eontract, the uncertainty or amount of risk that a contraet presents 
and the eontraet type. For more information about transaction costs see “Applying 
Insights from Transaction Cost Economics (TCE) to Improve DoD Cost Estimation” 
(Angelis et al. 2007). 

Eor an MDAP, there are many effeetive ways to reduee eoordination and 
motivation eosts: 

• Reduce eomplexity 

• Reduee uncertainty 

• Inerease measurement and monitoring 

• Aetively deter bad behavior 

• Mitigate uncertainty from asset speeificity with incentives 

• Increase eontestability 

A better understanding of transaction costs may contribute to improving eost 
estimates and redueing the number of MDAPs whieh experienee a eost breaeh. 


8 




D. ASSESSING TRANSACTION COSTS 


Transaction costs are difficult to assess beeause they are not easily identified and 
seldom captured in the aeeounting records. Therefore, a proxy must be used to measure 
the effect of transaction costs due to the lack of data. Angelis, et.al. have suggested the 
use of the SE/PM ratio as such a proxy. The SE/PM ratio is defined as SE/PM Cost 
divided by Total Program Cost, Equation 1.1. 


SE / PM Cost Ratio 


SE / PM Costs 
Total Cost 


( 1 . 1 ) 


The SE/PM cost ratio seems an appropriate proxy for transaction costs because 
program management costs and transaction costs are both defined, in part, as costs 
associated with eontract management (i.e., eost, schedule) and performance measuring 
and monitoring. Due to differences in program aeeounting practices and varying 
interpretations for elassifying eosts as Systems Engineering or Program Management, it 
is prudent to eollect the sum total of program SE/PM costs to ensure that the eorrect eosts 
are included in the data that is gathered. 

This study will analyze the SE/PM cost ratio, as a proxy for transaction costs, of 
MDAPs and the potential correlation between the SE/PM ratios and cost breaches. 
Determining the nature of any potential relationship between the SE/PM cost ratio and 
the number of cost breaches experieneed by a program would test the hypothesis that 
programs with higher SE/PM eost ratios will experience more cost overruns than 
programs with lower SE/PM eost ratios. It is presumed that programs with higher SE/PM 
cost ratios are in riskier contraetual relationships and have higher transaetion costs and 
will experience more eost overruns than those programs with lower SE/PM cost ratios. 
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II. COST BREACHES IN MAJOR DEEENSE ACQUISITIONS 

PROGRAMS 


Reining in cost growth for MDAPs has been problematic to the Department of 
Defense (DoD) for many years (Arena et al. 2006). The establishment of Selected 
Acquisition Reports (SARs) in 1967 was an early attempt to better monitor MDAP 
performance. SARs contain critical MDAP performance, cost and schedule information 
for use within the DoD and the Congress.^ Cost threshold breaches, by program 
appropriation category, are quickly and easily identifiable within the SAR. If a program 
cost threshold breach occurs, a brief explanation of how or why the cost threshold breach 
occurred is also found within the SAR. The default threshold value for a cost breach to 
occur is 10% cost growth. The various types of cost breaches and their cost appropriation 
categories are explained later in this chapter. 

In 1981, Senator Samuel Nunn and Congressman David McCurdy introduced the 
Nunn-McCurdy Amendment (10 U.S.C. § 2433 2006) in an effort to curb MDAP cost 
growth by holding the Department of Defense publicly accountable to Congress for their 
fiscal management of MDAPs. The Nunn-McCurdy Amendment became law with the 
FY 1983 Department of Defense Authorization Act, which establishes consequences for 
MDAPs which exceed cost thresholds. These consequences include that the Secretary of 
Defense provide Congress with program cost estimates using original and current 
requirements, as well as certification to Congress that the unit costs are reasonable and 
that the management structure of the program is adequate to control future costs 
(Government Accountability Office 2009a, 310-311). 

A. TYPES OF COST BREACHES IN DOD 

This thesis focuses on six important categories of appropriations where cost 
breaches often occur: acquisition procurement unit cost (APUC); program acquisition 
unit cost (PAUC); procurement; research, development, testing and evaluation (RDT&E); 

^ The information found in the SAR provides a “quiek look” summary of the MDAP’s ability to meet 
eost, performanee and sehedule objeetives and remain within the thresholds agreed upon by the Program 
Manager and Defense Aequisition Exeeutive. 
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military construction (MILCON); and acquisition-related operations and maintenanee 
(O&M). These categories of appropriations were selected due to their availability in the 
Defense Aequisition Management Information Retrieval (DAMIR) database. APUC and 
PAUC were selected for their relevance to cost overruns as defined by Nunn-MeCurdy 
cost breaches. Proeurement and RDT&E appropriations are often identified in the SAR as 
the appropriation whieh incurred the eost threshold breaeh, thus providing a logical 
explanation for their inclusion in the data set. MILCON and aequisition-related O&M 
appropriations, though infrequently identified as appropriations that breaehed a cost 
threshold, were also included in the data because they are components of the program 
aequisition cost. 

The following definitions are from the Defense Aequisition University (DAU) 
Glossary and the DAMIR SAR Data Entry Instructions. 

1. Acquisition Procurement Unit Cost (APUC) 

APUC is the unit eost that equals the program acquisition cost (the sum of all 
procurement funds) divided by the total number of fully eonfigured end items to be 
procured. Eor unit eost reporting and APB breach purposes, the APUC is calculated in 
Base-Year dollars (DAMIR 2011, 9-11). Base-year dollars are caleulated by using the 
currently approved acquisition program baseline (APB).^ 

2. Program Acquisition Unit Cost (PAUC) 

PAUC is the unit cost that equals the total program aequisition eost (as previously 
defined) divided by the program aequisition quantity (i.e., the total number of fully 
configured end items). Eor unit cost reporting and APB breaeh purposes, the PAUC is 
calculated in Base-Year dollars (DAMIR 2011). Base-year dollars are calculated by using 
the currently approved APB. 

^ The acquisition program baseline (APB) is the baseline that reflects the threshold and objective 
values for the minimum number of cost, schedule, and performance attributes that describe the program 
over its life cycle (Defense Acquisition University 2011b). In the event of a cost, schedule or performance 
objective breach, the APB can be revised; this is called “rebaselining” the MDAP. The initial APB is 
referred to as the “original” baseline and for a rebaselined program the revised APB is called the “current” 
baseline. More information about the rebaselining process can be found in Department of Defense 
Instruction 5000.02, Enclosure 4, Table 6. 
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3. 


Procurement 


Procurement appropriations fund those aequisition programs that have been 
approved for production (to include Low-Rate Initial Production (LRIP) of acquisition 
objective quantities), and all costs integral and neeessary to deliver a useful end item 
intended for operational use or inventory upon delivery (Defense Aequisition 
University 2011b). 

4. Research, Development, Testing and Evaluation (RDT&E) 

RDT&E appropriations fund the efforts performed by eontractors and government 
activities required for the Research and Development (R&D) of equipment, material, 
eomputer application software, and their Test and Evaluation (T&E) including Initial 
Operational Test and Evaluation (lOT&E) and Eive Eire Test and Evaluation (EET&E). 
RDT&E also funds the operation of dedicated R&D installation aetivities for the eonduct 
of R&D programs (Defense Aequisition University 2011b). 

5. Military Construction (MILCON) 

MIECON appropriations fund major projects such as bases, sehools, missile 
storage facilities, maintenance facilities, medical/dental clinics, libraries, and military 
family housing (Defense Aequisition University 2011b). Military construction costs will 
include only those projeets that directly support and are uniquely identified with the 
subject program (DAMIR 2011). 

6. Acquisition Operations and Maintenance (O&M) 

Acquisition O&M appropriations fund expenses sueh as civilian salaries, travel, 
minor eonstruction projeets, operating military forees, training and education, 
depot maintenanee, stoek funds, and base operations support (Defense Aequisition 
University 2011b). Acquisition-related operation and maintenance eosts may include 
aequisition eosts whieh, in special cases, have been funded by operation and maintenance 
(DAMIR 2011). 
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B. REPORTING COST BREACHES IN DOD 


As previously mentioned, the Selected Acquisition Report (SAR) provides a 
status update of MDAP cost, schedule and performance information to Congress. The 
document contains 18 sections which provide extensive details with which decision makers 
can make informed recommendations for the future of the program. In 1969, the SAR was 
used to provide periodic updates to the Senate Armed Services Committee. Beginning in 
1975, SARs were required to be submitted to Congress in order to keep Congress informed 
of the status of MDAPs (DAMIR 2011). In the 1980s, more federal legislation was 
introduced to increase visibility into DoD programs and control cost growth. More 
discussion about the DAMIR database and SARs is contained in Appendix B. 

The Nunn-McCurdy amendment (Title 10 U.S.C. § 2433 2006) to the Department 
of Defense Authorization Act of 1982 established congressional oversight of MDAPs that 
experienced cost growth above certain thresholds. The perceived effectiveness of the 
Nunn-McCurdy amendment led to it becoming a permanent requirement since 1983 in 
the Department of Defense Authorization Acts (Blickstein et al. 2011, 12). Nunn- 
McCurdy legislation is the current federal law used to control MDAP cost growth. There 
have been changes to the Nunn-McCurdy Act over the years, most notably in FY 2006 
and FY 2009. In 2006, the original baseline estimate was established as a threshold for 
measuring cost growth. This prevents a program from rebaselining to avoid a Nunn- 
McCurdy cost threshold breach. In 2009, Congress endorsed a requirement that any 
program with a critical breach (defined below) is presumed terminated unless that 
program has been certified by the Secretary of Defense. More information about MDAP 
certification requirements can be found in Public Law 111-23, “Weapon Systems 
Acquisition Reform Act of 2009,” May 22, 2009; DoDI 5000.02; and Defense 
Acquisition Guidebook, Chapter 10. 

The determination to use the original baseline or current baseline cost estimate for 
a program resides in whether the MDAP has been officially rebaselined. Early in the 
acquisition phase when the “original” or baseline cost estimate for an MDAP—also 
known as the APB—is established there may be a considerable amount of uncertainty 

about future project requirements. The uncertainty about various project requirements 
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may result in a program exceeding or “breaching” its original cost estimate. A cost 
threshold breach is considered to occur when any program cost expenditures exceed the 
APB. For the purposes of this report, a cost breach is considered greater than or equal to 
10% above the APB. Cost breaches that are greater than or equal to 15% above the APB 
are classified as Nunn-McCurdy cost breaches and have special reporting requirements 
that are not applicable to cost breaches which do not cross the Nunn-McCurdy threshold. 

The Nunn-McCurdy mandate requires reporting of specified cost information and 
states that “a Nunn-McCurdy breach is any increase in the Current Estimate (CE) of the 
Program Acquisition Unit Cost (PAUC) objective or Average Procurement Unit Cost 
(APUC) objective of 15% or more compared to the currently approved [acquisition 
program baseline] APB, or 30% or more compared to the original APB” (DAMIR 2011). 
Eurthermore, an increase of 15% or more against the current APB objective, or 30% or 
more against the original APB objective is a “significant” Nunn-McCurdy breach, and an 
increase of 25% or more against the current APB or an increase of 50% or more against 
the original APB or revised original APB is a “critical” Nunn-McCurdy breach. The type 
of cost threshold breach and the APB baseline that it is compared against is shown in 
Table 1. The APB, established by DoD components and approved by the Milestone 
Decision Authority, is the baseline cost estimate. 



APB Breach 
(RDT&E, 
Procurement, 
MIECON, O&M) 

Nunn-McCurdy 

“Significant” 

Breach 

Nunn-McCurdy 

“Critical” 

Breach 

Current Baseline 
Estimate 

10% 

+15% 

+25% 

Original Baseline 
Estimate 

N/A 

+30% 

+50% 


Table 1. Nunn-McCurdy Cost Breach Thresholds 
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Nunn-McCurdy cost threshold breaches, or eost overruns, are based on original 
cost estimates for PAUC and APUC at project completion and in the case of a program 
which has rebaselined, eost threshold breaehes are also based on the eurrent (i.e., 
rebaselined) eost estimate for PAUC and APUC at project completion. To better 
understand cost overrun ealculations. Figure 1 illustrates how the DoD uses earned value 
management (EVM) to eombine “government management requirements and industry 
best practices to ensure the total integration of cost, schedule, and work seope aspects of 
contracts” (Defense Aequisition University 2011a, 43-44). For more information about 
EVM, view the EVM Implementation Guide; http://www.aeq.osd.mil/evm/. 

Eigure 1 shows the total allocated budget of the contract, which is the sum of all 
of the budgets for work on a contract and the management reserve, a budget whieh is 
withheld by the program manager for risk management purposes. The budgeted cost of 
work performed (BCWP), budgeted cost of work currently scheduled (BCWS) and actual 
cost of work performed (ACWP) for a contract are the three curved lines in the figure. 
The BCWP represents the sum total cost of the work paekages that are open or eompleted 
for a speeified amount of time and the BCWS represents the sum total eost of the work 
that was budgeted for completion within a speeified amount of time. The BCWS line is 
also the eontraet’s performanee measurement baseline (PMB) and represents the 
contraet’s overall time-phased budget plan. The ACWP is the sum of actual costs that 
have been ineurred to aceomplish the work performed within a speeified amount of time. 
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Figure 1. Earned Value Management (From Defense Aequisition University 2012) 

The Estimate at Completion (EAC) is the sum of the ACWP and the estimate to 
eompletion (ETC) for the remaining work. The ETC ean be ealeulated using previously 
defined terms, the eost performanee index (CPI) and the sehedule performanee index 
(SPI). The CPI is ealeulated by dividing BCWP by ACWP and is a measure of eost 
effieieney. The SPI is ealeulated by dividing BCWP by BCWS and is a measure of 
sehedule effieieney. The formula for ealeulating ETC is: 

ETC ^{BAC-BCWP)/{CPI*SPI) (2.1) 

When the EAC, a eost estimate for the entire eontract, is a higher eost than the 
Budget at Completion (BAC), the established baseline eost estimate of the eontraet, a 
eost overrun is projeeted. At “Time Now” in Figure 1 the eontraet is not performing as 
well as it was predieted to perform. The eost for ACWP exeeeds the eost for BCWP, thus 
ereating a negative cost variance and driving the EAC above the BAC. Conversely, if the 
EAC is less than the BAC a positive cost variance is created and a cost underrun is 
projected since the contract is not expected to expend the entire allotted budget. Prior to 
project completion, to calculate the expected cost overrun the current cost estimate must 
be revised to incorporate ACWP costs and requirements modifications to the existing 
contract. This revised EAC is then compared to the BAC, the acquisition program 


17 















baseline (APB, as explained earlier in this ehapter) aequisition eost estimate for the 
eontract. The exaet pereentage of the eost overrun, based on the budget that was 
established at the APB, ean be ealeulated using the following equation: 


% Cost Overrun = 


EAC 

BAC 


XlOO 


100 


( 2 . 2 ) 


Figure 1 also illustrates the eost and schedule overruns for a contract or series of 
contracts at project completion. At “Completion Date” the contract should be complete, 
however the schedule variance (the difference between BCWP and BCWS) which often 
accompanies cost overruns causes the projected date to slip to the right-most dashed line. 
At this revised completion date, the cost difference between ACWP (no longer EAC 
because work is complete) and BAC is determined and this delta defines the size of the 
cost overrun. All of the scheduled work has been performed so BCWS will equal 
BCWP.^ The cost overrun calculation can be made using a similar formula: 


% Cost Overrun = 


ACWP 

BAC 


XlOO 


100 


(2.3) 


Having considered the categories of cost breaches for MDAPs, as well as exactly 
how cost breaches are calculated, it is necessary to understand where the cost breach 
information resides for DoD MDAPs. A single-source document streamlines the process 
for quickly identifying cost threshold breaches for MDAPs: the selected acquisition 
report (SAR). 


C. SELECTED ACQUISITION REPORTS (SAR) 

The Defense Acquisition Management Information Retrieval (DAMIR) system 
provides electronic Selected Acquisition Reports (SARs) in PDF format for active and 
inactive MDAPs. SARs contain threshold breach data and a brief explanation delineating 
the cause of the cost breach. A snapshot of the “Threshold Breaches” section (Table 2) of 


^ In many instances, contracts exhibit a relation between cost and schedule performance. For example, 
program managers may increase the man-hours allocation of a project in order to meet a schedule deadline. 
An increase in man-hours allocation, and particularly if the work effort qualifies as overtime pay, may 
cause project costs to increase. 
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the SAR provides this information at a glance. The SAR is a single report which contains 
details of critical parameters of an MDAP or major automated information system (MAIS), 
to include: the responsible office, threshold breaches, schedule, performance, current 
contracts and cost details. SARs are required to be submitted on an annual basis and in 
accordance with the SAR Data Entry instruction quarterly SARs are sometimes required. 


Threshold Breaches 


APB Breaches 

Schedule 

r 

Performance 

r 

Cost RDT&E 

r 

Procurement 

r 

MILCON 

r 

Acq O&M 

r 

Unit Cost PAUC 

r 

APUC 

r 

Nunn-McCurdy Breaches 

Current UCR Baseline 

PAUC 

None 

APUC 

None 

Original UCR Baseline 

PAUC 

None 

APUC 

None 


Table 2. Threshold Breaches Report from SAR 


SARs provide a snapshot of overall MDAP performance and not individual 
contract performance. For SARs which list a single contract, it is simple to deduce that 
the information reported in the SAR is the product of the listed contract. However, 
MDAPs typically require several contracts to be executed, often concurrently, and in 
many instances multiple contracts are listed on a single SAR. The current contract section 
of the SAR only includes “the six largest, currently active contracts (excluding 
subcontracts) that exceed $40 million in Then-Year dollars” (DAMIR 2011). Given that 
the standard user has limited database permissions and access, the granularity required to 
determine specific contract performance within an MDAP cannot always be achieved. 
This means that for MDAPs listing more than one contract in a SAR, if there is a 
threshold breach it is often difficult to determine which contract(s) caused the threshold 
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breech. On occasion, however, contract notes and comments will provide amplifying 
information about which contract(s) caused the threshold breach. 

To more accurately determine which contracts were responsible for threshold 
breaches, MDAPs which listed only one contract in the “Contracts” section of the SAR 
were selected. To further simplify the contract selection process, most programs which 
featured contracts with “effort numbers” were excluded. The difficulty with the effort 
numbers is that they complicate contract correlation across databases. For one contract, 
the effort number in DAMIR does not always accurately correspond to the contract effort 
number in the DCARC database. 

Per the promulgated guidance, “SAR termination will be considered when 90% of 
expected program deliveries or 90% of planned acquisition expenditures have been made, 
or when Selected Acquisition Reporting criteria are no longer met” (DAMIR 2011). 
Therefore, the active MDAPs found in DAMIR are between the beginning of the 
Engineering and Manufacturing Development phase (Milestone B) and 90% complete. 
Inactive MDAPs that are complete or >90% complete are no longer required to submit 
SARs. There are many instances where the final SAR for an MDAP indicates that the 
program is >90% complete, but not 100% complete. The use of data which may only be 
90% complete is presumed acceptable based on the following assumptions: 

• 90% of delivered product probably accounts for more than 90% of 
expenditures. 

• Experience indicates that after 90% of the budget has been expended, the 
remaining 10% of the expenditures have already been committed. 

SARs provide relatively high-level contract cost information. Eor this reason, it 
was decided that DAMIR would be used solely for the threshold breach data contained in 
the SARs. The program cost data found in the DD Eorm 1921 (Cost Data Summary 
Report, CDSR) provided in the Defense Cost and Resource Center (DCARC) database 
contains significantly more contract detail. DCARC, an entity of the Office of the 
Secretary of Defense Cost Assessment and Program Evaluation (OSD CAPE) 
organization, collects current and historical cost and resource data for MDAPs and 
MAISs. This cost and resource data is used by government cost analysts to develop cost 

estimates for government programs (Cost Assessment and Program Evaluation 2013b). 
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While it is normally not possible to determine system engineering and program 
management costs from an MDAP SAR, the work breakdown structure (WBS) format of 
the CDSR makes obtaining this information simple. The data extracted from the MDAP 
SARs and included in this report is located in Appendix D. 
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III. SYSTEMS ENGINEERING AND PROGRAM MANAGEMENT 

(SE/PM) COST RATIO 

This chapter briefly explores transaetion eost eeonomies in order to introduee the 
independent variable, the systems engineering/program management (SE/PM) eost ratio. 
After explaining how the SE/PM eost ratio is ealeulated, the souree of the raw data that is 
needed to make the ealculations, the Cost Data Summary Report (CDSR) (loeated in the 
Defense Automated Cost Information System [DACIMS]), is discussed. 

A. TRANSACTION COSTS 

Transaetion eost eeonomies (TCE) deseribes the eeonomie theory behind eost 
analysis performed by an institution in making the deeision of whether or not to 
partieipate in a market and ultimately incur or avoid expenses. It is generally accepted in 
the eeonomies community that there are three primary eategories of transaction costs: 
search and information costs; bargaining and decision costs; and policing and 
enforcement eosts (Johnson 2005). To illustrate these cost categories, eonsider the non¬ 
monetary exehanges whieh oeeur during the marketing and purehase of a home. The 
prospeetive homeowners are working with a realtor to find an aeeeptable home based on 
its physieal loeation, size and other intrinsic characteristics (search and information 
costs). The prospeetive homeowners will then negotiate between their realtor and the 
sellers’ realtor to determine a purehasing price for the home (bargaining and deeision 
eosts). East, if the home was purehased using money that was borrowed from a mortgage 
lender, the mortgage eompany will ensure that the buyers are upholding their end of the 
agreement by paying the monthly mortgage bill (policing and enforcement eosts). 
Clearly, there is a real (time and effort) eost assoeiated with these three types of 
transaetion eosts, albeit diffieult to quantity. Eirms and eorporations eonsider TCE, if not 
by name then by praetiee, to guide their deeision to outsouree or vertieally integrate a 
serviee or produet whieh they reeeive. 

The previous paragraph establishes that transaetion eosts are assoeiated with 
“souree seleetion, periodie eompetition and renegotiation, eontraet negotiation and 
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management, performanee measuring and monitoring and dispute resolutions” (Angelis 
et al. 2008). Although they are not available directly in standard databases, one way to 
estimate them is by using SE/PM costs. (The importance of those costs in any given 
program could then be estimated with the fraction of total costs allocated to SE/PM.) As 
defined in MIE-STD-881C, the combined definitions of systems engineering costs^ and 
program management costs^*’ are representative of transaction costs and therefore make 
SE/PM costs a logical proxy for transaction costs. 

The SE/PM cost ratio for a program (the ratio of SE/PM costs divided by total 
program costs as shown in Equation 3.1), specifies the proportion of total program costs 
that are dedicated to managing, integrating and directing the program: 


SE / PM Cost Ratio 


SE/PM Costs 
Total Cost 


(3.1) 


This report hypothesizes that a higher SE/PM cost ratio is predictive of future cost 
threshold breaches. The higher the SE/PM cost ratio then the more risky a transaction is 
considered—the converse is also implied. Categorizing a SE/PM cost ratio as high or low 
is a judgment call. There does not appear to be a directed standard or normal-practice 
SE/PM cost ratio definition across MDAPs. One reason that the SE/PM cost ratios may 
vary is due to the type of weapon system. Another reason SE/PM cost ratios differ 
between MDAPs, and sometimes across contracts within the same MDAPs, is due to 
subjective interpretations among contractors (and program managers) about the definition 
of SE/PM costs and non-uniform standards regarding which costs qualify to be 
categorized as SE/PM costs (Stem, Boito and Younossi 2006). 

The SE/PM cost values used in this thesis are extracted from the WBS line item 
values for “To Date” SE/PM Cost and EAC SE/PM cost which are listed on the Cost 
Data Summary Report (CDSR), DD Eorm 1921. The “To Date” SE/PM cost is the sum of 


^ Systems engineering costs are the technical and management efforts of directing and controlling a 
totally integrated engineering effort of a system or program (Department of Defense 2011, 221). 

10 Program management costs are costs associated with the business and administrative planning, 
organizing, directing, coordinating, controlling, and approval actions designated to accomplish overall 
program objectives, which are not associated with specific hardware elements and are not included in 
systems engineering (Department of Defense 2011, 222). 
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recurring and non-recurring costs for the SE/PM WBS line item as of the date of the 
WBS report. The EAC SE/PM cost is the projected SE/PM cost at contract completion. 
The SE/PM costs are inclusive of the total contract costs less the contractor’s profit/loss 
or fees. Eor clarity, a sample WBS with SE/PM cost ratio calculations is provided in 
Appendix C. 

We analyze SE/PM costs over the life of the MDAP for trends. These trends are 
significant because the year-to-year increase or decrease in SE/PM costs over the life of 
the MDAP may be a proactive or reactive measure by MDAP program managers and/or 
contractors to avoid a threshold breach. Perhaps, an MDAP with a gradually increasing or 
decreasing SE/PM cost ratio may be indicative of a future cost threshold breach or 
prevention thereof. 

The numerator of the SE/PM cost ratio is the amount of SE/PM cost expenditures 
and the denominator is MDAP expenditures or total costs. Eor completeness of analysis, 
the “To Date” SE/PM and total costs, as well as the EAC SE/PM and total costs are 
recorded. These four numbers are necessary for the “To Date” SE/PM cost ratio and EAC 
SE/PM cost ratio calculations, respectively. To illustrate this point, according to our 
analysis a common sense approach for evaluating the cost performance of MDAP Project 
X would be to observe its current or “To Date” SE/PM cost ratio and its predicted or 
EAC SE/PM cost ratio. 

B. COST AND SOFTWARE DATA REPORTING 

The MDAP database found in the Defense Automated Cost Information System 
(DACIMS) 3.5 Eibrary and discussed further in Appendix C, electronically lists the 
recorded contracts for most MDAPs. This Cost and Software Data Reporting (CSDR) 
library contains folders of active and inactive MDAP contract data sorted by weapons 
system types. More specifically, contract information within each of these folders 
includes: the Cost Data Summary Report (CDSR), DD Eorm 1921; the Eunctional Cost- 
Hour Report, DD Eorm 1921-1; and the Contract Work Breakdown Structure (CWBS) 
Index and Dictionary among other documents. Erom the CDSR that is found in 
DACIMS, we can extract the line item costs that we need for the SE/PM cost ratio 
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calculation. Within a single contract, it is not unusual to find cost data for both the prime 
vendor who retains the listed eontraet and subeontraeted vendors who support the 
eontraet via the prime vendor. To simplify the data eolleetion proeess, only the eost data 
provided in the prime eontraet by the primary vendor is reeorded. 

A eomplieation to the data eolleetion proeess oeeurs when the list of eontraets for 
an MDAP in the DACIMS library does not eompletely mateh the list of eontraets for the 
same MDAP in its SAR (Seleeted Aequisition Report) whieh is loeated in the Defense 
Aequisition Management Information Retrieval (DAMIR) database. For this report, 
MDAPs with eost threshold breaehes are exeluded from the data set if the listed eontraets 
are not available in both databases, DACIMS and DAMIR. This is beeause we need to 
know the SE/PM eost ratios for the MDAP during the year when the eost threshold 
breaeh oeeurred. 

For an MDAP whieh reports no eost threshold breaehes, it is not neeessary for all 
of the listed eontraets to be available in both databases. The reason for this is that relianee 
on the SAR in the DAMIR database assures that no eost breaehes oeeurred. Trusting the 
SAR allows reeording of the SE/PM eost ratios for the eontraets that are available in the 
DACIMS database without fear of exeluding a eontraet SE/PM eost ratio for a year that 
eontained a eost threshold breaeh—sinee the SAR eonfirms that no eost threshold 
breaehes oeeurred. 

In many instanees the eost data between DAMIR and DACIMS do not mateh 
exaetly. Cost data as reported in the SAR is from the perspeetive of the government 
representative, or program manager. Cost data eontained in the CDSR is reported by the 
eontraetor. The differenee between government and eontraetor eost ealeulations may 
aeeount for some of the disparity between the two sets of reported eost data. Another 
souree of disparity is that the CDSR and SAR are normally submitted aeeording to 
different timelines. To lessen aeeounting ineonsisteneies, all eost data were retrieved 
from the DACIMS library (CDSR) and the threshold breeeh data were obtained solely 
from the DAMIR database (SAR). Although the aeeounting figures in the two databases 
are different, it is assumed that the differenees are not so signifieant that they would 
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change the status (breach or no breach) of any MDAP in regard to its cost threshold 
breaeh criteria as reported in the SAR. 

The potential for inconsistency in SE/PM cost determination across MDAPs 
exists due to the individual MDAPs assigning (perhaps implieitly or unintentionally) 
idiosyncratic definitions for SE/PM costs within their programs. Some insight as to how 
MDAPs define various eontract eosts is located in the CWBS Index and Dictionary. The 
CWBS dictionary contains a description of the technieal and cost data for the CWBS 
elements within the contract. By utilizing the CWBS it is possible, albeit tedious, to 
ensure to a high degree of confidenee that the SE/PM costs between contracts in this 
study are inclusive of the same types of WBS elements. 

Contract data found in the CDSR is given as an initial, interim or final cost 
summary and is submitted in accordance with the eontract CDSR plan as approved by the 
Office of the Seeretary of Defense Deputy Director, Cost Assessment (OSD DDCA). 
This means that the frequeney of submission for eost data doeuments aeross different 
MDAPs varies considerably and that contractor cost data may not be submitted on an 
annual basis. Eor this report, to ensure that the eost data contained in the CDSR for an 
MDAP is relevant to the period covered by the same MDAP’s SAR, the reporting periods 
for the two documents are verified to be within the same ealendar year. Eor example, if 
an MDAP’s CDSR is dated July 2010 and the associated SAR is dated December 2010 
then this program and its data are eligible for inclusion in the data set. 
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IV. METHODOLOGY OF ANALYSIS FOR COST BREACHES 

AND SE/PM COST RATIO 

This chapter presents the methodology used to test the hypothesis that the 
likelihood of a major defense aequisition program (MDAP) cost threshold breach may be 
assoeiated with the transaction costs of the MDAP. While the system engineering/ 
program management (SE/PM) cost ratio is a proxy for transaction costs, interactions 
between other faetors which might be related to cost breaches are acknowledged and 
explained using an influence diagram. Data were eollected for SE/PM eost ratio 
determinations, as well as for some of the other factors that were presented in the 
influence diagram. This chapter presents a detailed discussion that describes how all of 
the data were arranged for use in the statistie models. Aecordingly, an explanation of the 
ordinary least squares (OES) and logistic regression models whieh were used for the 
analysis in this report are ineluded in this chapter. 

A. HYPOTHESIS 

Numerous reports (e.g., Bolten et al. 2008) have studied cost growth in 
Department of Defense (DoD) programs, yet researeh that seeks to establish a potential 
correlation between the SE/PM cost ratio of an MDAP and the oecurrence of a cost 
breach is limited (Angelis et al. 2008). Is a high SE/PM cost ratio associated with an 
aequisition program baseline (APB) or Nunn-McCurdy breaeh? What is considered a 
high SE/PM ratio? A recent RAND study established that MDAP SE/PM costs vary 
between programs depending on the program type.n Eor instanee, the average SE/PM 
costs for aircraft development programs are different from the average SE/PM costs of 
weapons programs. 

This report seeks to determine the nature of any potential relationship between 
transaction costs (using the SE/PM cost ratio as a proxy) and the likelihood of cost 
breaches experienced by a program. The hypothesis is that programs with higher SE/PM 

11 SE/PM cost is variable between MDAPs because program managers are idiosyncratic with what is 
included in SE/PM costs. There is no generally accepted standard of what qualifies as SE/PM costs, 
although it is defined in MIL-STD-881C (October 2011) (Stem, Boito and Younossi 2006). 
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cost ratios are more likely to experience eost breaehes than programs with lower SE/PM 
eost ratios. This is based on the assumption that higher SE/PM eost ratios are related to 
riskier eontraetual relationships sinee more time, effort and resources are expended to 
meet performance and sehedule deadlines when eompared to less risky eontracts. 
Programs with higher transaetion eosts may experienee more eost breaehes as a result of 
those transaetion eosts not being aeeounted for in the original eost estimate. 

The influenee diagram in Eigure 2 describes the interaetions between factors 
which may be assoeiated with the oecurrenee of a eost breaeh. There are many other 
variables whieh affeet eost breaehes, but the variables identified in this influenee diagram 
(i.e., SE/PM eost ratio, program maturity and eontraet type) were seleeted beeause they 
ean be represented with the data eolleeted. Beginning with the maturity of the program, 
the maturity variable has quotation marks beeause “maturity” means different things to 
different readers. While in this report “maturity” is defined as the time in years sinee 
program initiation or Milestone B, many will think first about teehnology readiness levels 
(TRLs). Program managers and eontractors work together to manage away risk and 
eomplexity as the program ages. This proeess is eaptured by Eigure 2 and is later 
diseussed. Coneerning the risk and eomplexity of the MDAP, the program managers and 
eontraetors ean only state edueated opinions and informed assumptions to guide them in 
the negotiations for an appropriate eontraet type. In some eases, the risk assoeiated with 
the eomplexity of a new teehnology may result in a eost overrun. In other instanees, 
perhaps transaetion eosts assoeiated with managing the MDAP eontraets and business 
negotiations eontribute to eost overruns and eost breaches. Beeause transaction costs are 
diffieult to quantify, the SE/PM eost ratio can be used as a proxy to help understand the 
eontribution of transaetions eosts to a program’s eost overruns and subsequently the 
probability of the program having a eost breaeh. Of eourse not all eost overruns are 
eonsidered eost breaehes. Current estimates whieh exeeed the estimated eost of a 
program by any amount are eonsidered eost overrun situations. Cost breaehes result when 
the amount of the eost overrun exeeeds eertain parameters defined by regulation. In this 
thesis, a default value of 10% eost overrun eonstitutes the eost breaeh threshold. Einally, 
the dashed lines in Eigure 2 represent factors that one eannot observe or does not know 
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how to observe and must be dealt with qualitatively or by using proxies. The solid lines 
represent factors that can be quantitatively evaluated. 



Figure 2. Influence Diagram 


There are two types of cost breaches; APB and Nunn-McCurdy breaches. In 
order for a program to incur an APB breach or Nunn-McCurdy breach, currently 
estimated program expenditures must be greater than the APB estimate at completion by 
at least 10% or 15%, respectively. (Cost breaches are discussed in detail in Chapter II, 
Section B.) The influence diagram shows that not all cost overruns are cost breaches by 
associating a probability with APB or Nunn-McCurdy breaches. The desire to determine 
the probability of a binary outcome (breach or no breach) occurring in MDAPs led to the 
decision to use logistic regression (logit) to model and analyze the data. 

This report takes steps toward developing a logistic regression model which 
program managers may be able to utilize to determine whether their MDAP is in jeopardy 
of incurring a cost threshold breach. According to this model, program managers will be 
able to estimate the probability of their program incurring an APB or Nunn-McCurdy 
breach based on the SE/PM cost ratio which their program maintains. Three models were 
used to assess the impact of an increase in SE/PM cost ratio on the likelihood of a 
program experiencing an APB or Nunn-McCurdy breach: a pooled OLS regression, a 
fixed effects model and a population averaged model. 
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The null hypothesis of this research states that better incorporation of transaction 
costs into cost estimates will not significantly affect the ability of an MDAP to operate 
within its approved financial constraints as measured by cost breaches. 

B. DATA 

The cost data for 32 MDAPs were collected to determine the presence of any 
correlation between MDAP cost breaches and transaction costs. The MDAPs were not 
randomly selected, but instead selected based on the availability of relevant data from the 
two databases used, Defense Acquisition Management Information Retrieval (DAMIR) 
and Defense Cost and Resource Center (DCARC). Cost threshold breach information, 
categorized by the cost appropriation that sustained the cost breach, was available from 
the Selected Acquisition Report (SAR) in the DAMIR database. At a minimum, the 
MDAP SARs were generated annually as a means for the program manager to provide a 
status report to Congress. The line item cost inputs for calculating the “to date” and EAC 
SE/PM cost ratios were obtained from the Cost Data Summary Reports (CDSRs) found 
in the Defense Automated Cost Information System (DACIMS) library located within the 
DCARC database. The 32 MDAPs researched represent the Departments of the Air 
Eorce, Army, and Navy, plus Joint programs. 

The MDAP data were collected and organized within a data table by dependent 
and independent variables across a time series for each MDAP (see Appendix D for a list 
of the MDAPs used in this research). Any type of cost breach, APB cost breach or Nunn- 
McCurdy cost breach, is considered the binary-outcome dependent variable in this 
analysis: cost breach or no cost breach. There are three independent or explanatory 
variables that were included in the analysis for this report: EAC SE/PM cost ratio, time 
since program initiation and program contract type. While the exact nature of the 
relationship between cost threshold breaches and these explanatory variables is unknown, 
it is reasonable to suppose that the explanatory variables influence the cost performance 
of the MDAPs in some manner. 

This research uses binary, clustered and panel data. Prom the sample provided in 
Table 3, the binary nature of the data is shown to pertain to the dependent variable “Cost 


32 



Breach.” In simple terms, the program either sustained an APB or Nunn-McCurdy 
breach during a specific calendar year or it did not. The clustered nature of the data 
describes the various categories of program types that the data represents. For example, 
in Table 3 the MDAP A program may be categorized as a ship building program and the 
MDAP B program may be categorized as an aircraft development program. This is of 
particular importance because the average SE/PM cost ratio is variable between program 
categories and this grouping must be accounted for in any statistical analysis. 


Program 

Program 

Number 

Contract 

Year 

EAC 

SE/PM 

APB 

Cost 

Breach? 
(0 - no, 

1 - yes) 

N-M 

Cost 

breach? 
(0 - no, 

1 - yes) 

"Maturity" 

(years) 

Contract 

Type 
(0 - Fixed, 
1 - Cost) 

MDAP A 

20 

2005 

0.0441 

0 

0 

5 

1 

MDAP A 

20 

2006 

0.0318 

0 

0 

6 

1 

MDAP B 

21 

2007 

0.0144 

0 

0 

11 

0 

MDAP B 

21 

2009 

0.0641 

1 

0 

13 

0 

MDAP B 

21 

2010 

0.1626 

0 

0 

14 

0 

MDAP B 

21 

2011 

0.1342 

1 

1 

15 

0 

MDAPC 

22 

2000 

0.0292 

1 

0 

7 

0 


Tables. Sample Data Set 


The importance of accounting for clustering is noted in Logistic Regression Using 
the SAS System (Allison 2001, 180): 

• Ignoring clustering and treating each observation as independent may 
cause the calculated standard errors to be underestimated and the test 
statistics to be overestimated. 

• Applying conventional logit analysis, which will be explained in Section 
C of this chapter, to clustered data will produce standard errors that may 
be larger than those standard errors produced by other analysis methods. 

• Clustered data can be corrected for biases that may occur in any 
application of binary regression, i.e., a particular logit analysis method can 
correct for bias due to omitted explanatory variables. 
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Last, the dataset in Table 3 shows observations of multiple independent variables 
that have been eollected for a program over a time series (measured in years) and in 
statistics this type of data collection is known as panel data. 

Program work breakdown structure (WBS) line item costs for SE/PM were 
obtained from cost data summary reports (CDSR, DD Form 1921) found in the DCARC 
database. The SE/PM cost ratios calculated serve as the proxy for transaction costs. The 
“to date” SE/PM cost ratios were calculated by dividing the “to date” SE/PM cost by the 
“to date” total cost of the program. Similarly, the estimate at completion (EAC) SE/PM 
cost ratios were calculated by dividing the EAC SE/PM cost by the EAC total cost of the 
program. SE/PM cost data ratio calculations were recorded by year for each program, as 
available. During the development of this report, it was decided that “to date” SE/PM 
cost ratios provided extraneous information and would not be included in data analysis 
since MDAP cost breaches are based on EAC cost projections and not “to date” cost 
expenditures. 

In this report, program “maturity” describes the time that has elapsed since 
program initiation (or Milestone B, the entry point into the Engineering and 
Manufacturing Development phase). The concept of maturity may seem quite simple; 
however, this definition also implies attainment of the DoD requirements for an MDAP 
to accomplish Milestone B approval, namely that the design and technology utilized in 
the project are mature. 12 Using the DoD Milestone B requirements to define maturity 
ensures that each program analyzed in this report is assessed beginning with a 
comparatively equitable baseline. Presumably, as time progresses, a project’s contracts 
should have a smaller likelihood of incurring a cost breach for each successive year after 
program initiation because there is more available information about the project and the 
realm of the unknown is decreasing. The increase in system or program information and 
the corresponding reduction in uncertainty are represented by the uncertainty-information 


12 Milestone B approval authorizes an MDAP to enter the Engineering and Manufacturing 
Development phase of the acquisition process. Statutory requirements for MDAPs to achieve Milestone B 
approval are found in Title 10 U.S.C. § 2366b. These requirements include that the program is certified by 
the milestone decision authority to be affordable, fully funded through the Future Years Defense Program 
(FYDP) and that the cost and schedule estimates are reasonable. 
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tradeolf as a function of program maturity as shown in Figure 3 (After Yoe 2000, 2). This 
figure suggests that the greatest uneertainty in a eost estimate oeeurs in the earlier stages of a 
program when there are more future unknowns as compared to the later stages of a program. 



Figure 3. Uneertainty and Information as a Function of Maturity. 

(After Yoe 2000, 2) 

The time since program initiation at Milestone B is obtained from the Seleeted 
Acquisition Report (SAR) that is promulgated annually in December by the MDAP 
program office and an extensive listing of the active and inactive SARs for all DoD 
programs (Air Force, Army, Joint, Navy) can be accessed via the Defense Aequisition 
Management Information Retrieval (DAMIR) database. The time elapsed sinee Milestone 
B for a eontract may not always aeeurately represent the amount of time that an MDAP 
has been in existenee. For instance, the AIM-9 Sidewinder first entered serviee the 1950s 
(Wikipedia 2013), but many of the later variants of the AIM-9 were developed using 
contracts from more recent years, such as the AIM-9X program which began in 1996 
(MacKenzie 1997). In instances such as the AIM-9X program, the program initiation date 
for the specific variant contracted was utilized to determine the maturity of the program. 

Program eontract type is obtained from the eontracts (CDSRs) whieh are found in 
the DC ARC database. Programs are noted as having either firm fixed price type contracts 
or cost-plus type contracts. It is presumed that riskier programs will have cost-plus 
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contracts since contractors will be hesitant to enter into firm fixed priee eontraets which 
require them to metieulously manage the eosts of the work in order to earn a profit. 13 

C. APPROACH TO ANALYSIS 

Given the nature of the data, a good starting point was to eonsider a bivariate 
model to explain eost breaehes and transaetion eosts. Using a standard bivariate 
regression, one variable would be eonsidered to prediet an outeome variable (Aeoek 
2006, 211). In the ease of this researeh, a high value for the independent variable of 
SE/PM eost ratio may manifest as a possible ex-ante indieator for a eost breaeh in an 
MDAP. It has already been established that we are analyzing elustered panel data and 
that the average SE/PM cost ratios are different between program types. Thus, a single 
SE/PM eost ratio value that would indicate a eost breach is of no use to a program 
manager sinee there is no aecounting for the varianee aeross program types. Eor example, 
the overall SE/PM eost ratio eould be a value that is lower than the average SE/PM eost 
ratio for programs that normally maintain a eomparatively high SE/PM eost ratio. 
Certainly SE/PM eost ratio alone does not indieate that an MDAP will have a eost breaeh 
(e.g., Sullivan 2011). While other factors that might impact cost breaches were taken into 
aceount, data elustering has not yet been taken into eonsideration. 

A multiple regression model is the next logieal model that may be eonsidered. A 
multiple regression makes sense beeause it would ineorporate input from many predietors 
or independent variables to prediet the outeome variable (Aeoek 2006, 212). Again, 
finding a single SE/PM eost ratio value whieh may indicate an inereased likelihood for a 
breaeh does not seem like it would be helpful to a program manager sinee SE/PM cost 
ratios variables between MDAP types. 


13 A cost plus contract is a type of contract that provides for payment to the contractor of allowable 
costs incurred in the performance of the contract, to the extent prescribed in the contract. This type of 
contract establishes an estimate of total cost for the purpose of obligating of funds and establishes a ceiling 
that the contractor may not exceed without prior approval of the Contracting Officer (CO). A firm fixed 
price contract provides for a price that is not subject to any adjustment on the basis of the contractor’s cost 
experience in performing the contract. This type of contract places upon the contractor maximum risk and 
full responsibility for all costs and resulting profit or loss. It provides maximum incentive for the contractor 
to control costs and imposes a minimum administrative burden on the government (Defense Acquisition 
University 2011b). 
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Perhaps instead of finding a single SE/PM eost ratio that would presumably alert 
program managers to the impending danger of an APB or Nunn-MeCurdy eost breaeh, it 
would be more helpful to provide program managers with an indication of the risk of 
incurring a cost breach. A ratio, adjusted to be program-specific, that program managers 
may reference for their decision making. 

In the case of an MDAP cost breach, the outcome variable is binary: breach or no 
breach. Logistic regression is a type of regression that was designed for binary-outcome 
dependent variables (Acock 2006, 250). Logistic regression predicts the probability of 
binary outcomes (breach or no breach). The S-curve seen in Ligure 4 indicates the 
relationship between the linear probability and logit models. In order to fit this model, a 
logit or maximum likelihood estimation is calculated for each observation. 
Mathematically, the logit value (Equation 4.1) is the natural logarithm of the odds ratio 
for a given event to occur (Acock 2006, 256); and the odds ratio is simply the ratio of the 
event occurring versus the ratio that the event will not occur. Each point on the logistic 
regression line in Eigure 4 represents a logit value which describes the relationship 
between the dependent and independent variable. 

Logit = In (odds ratio) (4.1) 

Although the probability of an MDAP crossing a cost threshold is not linearly 
related to the SE/PM cost ratio, the logit is and therefore the predicted logit of a cost 
breach occurring is the statistical equivalent to the predicted probability of a cost breach 
occurring (Acock 2006, 253). 
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Comparing the LP and Logit Models 

Y 



Figure 4. Linear Probability versus Logit Model 
(From http://www.appstate.edu/~whiteheadjc/service/logit/logit.gif) 

In this study, the pooled OLS regression serves as the baseline model. The data is 
considered “pooled” because it uses all of the independent data observations. 
Unfortunately, while the binary outcome is limited to a maximum probability of 1.0 
(100%) and minimum probability of 0.0 (0%), OLS regression is not constrained to these 
limits. Because it uses linear estimates, OLS regression may result in predicted 
likelihoods greater than 1.0 or less than 0.0. Additionally, OLS regression assumes that 
each observation is independent, but we already know that this is not the case with the 
MDAP data: observations from within the same cluster or program are probably more 
alike than observations from another cluster. In other words, it is expected that within 
group outcomes are related differently than the cross-group outcomes. The assumption 
that each observation is completely independent skews the calculated standard errors for 
the data. A solution to the problems presented by the OLS regression model is to employ 
one of several maximum likelihood estimators and panel data techniques. 

The second model tested was a fixed effects logistic regression model. The fixed 
effects or conditional logit model adjusts for bias introduced by omitted explanatory 
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variables within the elusters of data. It eontrols for oharaeteristies whieh may eause a eost 
breaeh even if those characteristics cannot be measured. The standard logit model is 
presented in Equation 4.2 (Allison 2001, 181), where Pit represents the probability that at 
a specific time an MDAP breaches a cost threshold, Pu is the vector of regression 
coefficient which represents the association of cost breaches and the explanatory variable 
to a unit change in the independent variable (SE/PM cost ratio) and x represents the 
explanatory variable for the respective MDAP. 

Log(P„/ 1-P„) = Ax (4.2) 

The fixed effects (conditional) logit model, which accounts for dependence of the 
observations within a group, modifies the basic logit model (Equation 4.3) (Allison 2001, 
188). In this equation, a represents a set of fixed constants for each MDAP in the group. 
This model measures the impact of deviations by the independent variable from the 
group’s (program’s) mean. This model can be considered to describe the impact of short 
run changes, i.e., that a change in the SE/PM cost ratio for an MDAP will change its 
probability of breaching a cost threshold. 

Log (P;, / l-P;,) = « + (4.3) 

Einally, the relationship between SE/PM cost ratio and cost breach was modeled 
using a population averaged logit model. Contrary to the fixed effects method that looks 
within programs for changes in the SE/PM cost ratio, the population averaged method 
looks across programs to measure changes in the SE/PM cost ratio. Therefore, using 
Equation 4.3 to explain the population averaged logit model, a represents a set of 
averaged constants for MDAPs across program groups. The population averaged method 
is most useful for measuring long run effects, i.e., measuring whether programs with high 
average SE/PM cost ratios are more likely to breach. 

D. SUMMARY 

A logistic regression model, or a maximum likelihood estimator such as logit 
which uses the natural logarithm of the odds ratio, is appropriate for this research because 
it was designed for use with binary-outcome dependent variables. Additionally, a fixed 
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effects logit model will adjust for bias introduced by omitted explanatory variables within 
the clustered panel data and a population averaged logit model is appropriate for 
examining across clusters of panel data. Now that the models used in this research have 
been justified, it is necessary to look at the results that the various logit models provide 
and the differences between them. 
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V. INTERPRETATION OE EAC SE/PM COST RATIO ANALYSIS 


This chapter contains a detailed analysis of the data and provides an interpretation 
of the analysis results. This first section contains a general description of the data 
collected. The next three sections describe the relationship between major defense 
acquisition program (MDAP) cost breaches and the independent variable and explanatory 
variables that incrementally expanded the model. These three sections provide equations 
that model the variables being tested, as well as the results of ordinary least squares 
(OLS) and logit regression analysis. The chapter concludes with a summary of the most 
significant data analysis results. 

A. OBSERVED MDAP EAC SE/PM COST RATIOS 

A visual representation of MDAP cost breaches compared to MDAP estimate at 
completion (EAC) system engineering/program management (SE/PM) cost ratios is 
shown in Eigure 5. As a reminder, in this report cost threshold breaches are cost overruns 
which are greater than or equal to the default APB cost breach threshold value of 10%. 
Therefore, for this model cost breaches include APB cost breaches and Nunn-McCurdy 
cost breaches, since the Nunn-McCurdy cost breach thresholds are all greater than 10%. 
More specifically, Eigure 6 shows the total number of cost breaches reported by the 
Selected Acquisition Reports (SAR) that a program has sustained and compares it against 
the program’s average SE/PM cost ratio. There are no discernible patterns in Eigure 6, so 
the data were re-configured to see whether another presentation yielded any patterns or a 
different interpretation. 
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Figure 5. MDAP EAC SE/PM Cost Ratio versus Cost Breaches 


Figure 5 is based on the cost threshold breach and corresponding cost data that 
was available for 32 selected MDAPs since 1998. From this representation of the data, it 
appears that more than half of the MDAPs maintain an EAC SE/PM cost ratio of 0.20 or 
less and have experienced less than two cost breaches since 1998. Furthermore, it can be 
inferred that most of the programs have experienced at least one cost breach and these 
inferences seems to confirm a recent RAND report which found that most MDAPs actual 
costs exceeded baseline cost estimates (Arena et al. 2006). 

Observations of the MDAP SE/PM cost ratios in this report seem to concur with 
the RAND study which suggests that trends in SE/PM costs vary across MDAPs (Stem, 
Boito and Younossi 2006). For this reason, it is challenging to identify relationships 
between SE/PM costs and cost breaches. The average EAC SE/PM cost ratio for the 
sample data set of 32 MDAPs is 0.16. This suggests that on average, program managers 
of the MDAPs studied in this report spent 16% of their budgets on systems engineering 
and program management activities. Recall that the systems engineering and program 
management costs are used as proxy measures of the transaction costs required to 
administer and to manage the MDAP. 

For most of the programs reviewed, actual costs exceeded the baseline cost estimate established at 
Milestone II (program initiation), as measured by the cost growth factor (Arena et al. 2006). 
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A histogram of the EAC SE/PM cost ratios observed in this study is shown in 
Figure 6. From this histogram we can observe that while most observed ratios varied 
between 0.05 and 0.35, there do appear to be two outliers having values of 0.50. 


EAC SE/PM Cost Ratio Histogram 



EAC SE/PM Cost Ratio 


■ Frequency 


Figure 6. EAC SE/PM Cost Ratio Histogram for MDAPs 

We assume for parametric analysis that the variables are normally distributed 
because one of the underlying assumptions of OES regression is that the data is normally 
distributed. A normal probability plot (Figure 7) was used to evaluate the normality of the 
data. This plot suggests that the data is normally distributed with an r-squared value of 
0.85.*^ While this normality will support the use of OES regression in our analysis, it is 
not required for logistic regression. 


The r-squared value measures the goodness of fit of the data to the trend line shown in Figure 7. In 
a perfect dataset, the normal probability plot of normally distributed data will be a perfectly straight line 
with an r-squared statistic of 1.0. The dataset used in this report is confirmed to be normally distributed 
due to its relatively linear normal probability plot and the r-squared statistic of the trend line being 0.85. 
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Normal Probability Plot 



Figure 7. Normal Probability Plot of EAC SE/PM Cost Ratio 

In Figure 8, we see MDAP EAC SE/PM cost ratios binned by various numbers of 
cost threshold breaches for selected MDAPs since 1998. It appears that the information in 
Figure 8 can plausibly be described by a simple linear regression and this was a primary 
reason that the linear bivariate and multiple regression models were suggested to model 
the data, as explained in the previous chapter. If this is a valid correlation, it supports the 
hypothesis that “programs with higher SE/PM cost ratios will experience more cost 
overruns than programs with lower SE/PM cost ratios.” Armed with this information, it 
is logical to suspect that the likelihood of a program sustaining a cost threshold breach 
could be associated with a certain SE/PM cost ratio. The null hypothesis is that the 
probability of a cost threshold breaches is not associated with the MDAP EAC SE/PM 
cost ratio and that the model has no explanatory power. 
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Trends in MDAP EAC SE/PM Cost Ratio 
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Figure 8. Trends in MDAP SE/PM Cost Ratios 

B. COMPARISON OF MDAP COST BREACHES TO SE/PM COST RATIOS 

The OLS and logistic regression techniques that were discussed in the previous 
chapter will be utilized and some of the strengths and weaknesses of these models will be 
described. The most valuable information obtained from these models is the marginal 
effects and average marginal effects values. The marginal effects values describe the 
impact of the independent variable on the dependent variable. For example, if our 
analysis were to conclude that the average marginal effect of EAC SE/PM cost ratio was 
0.95, then the model is stating that a 1.0% change in the independent variable (EAC 
SE/PM cost ratio) will result in a 0.95% change in the dependent variable (cost breaches). 
It is important to understand that although correlation may exist between the dependent 
and independent variable, causation between the two variables cannot be implied. 

We will now look at results yielded by the analysis methods that were described 
in the previous chapter: pooled OES regression, logit fixed effects regression and logit 
population averaged regression. In the first series of model runs, the variable being tested 
for correlation with cost breaches is the EAC SE/PM cost ratio. For the remaining 
sections of this chapter, it will be helpful to consider the following linear model to 
explore many variables and cost breaches: 
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Y = a + poXo + pXi. 


Variations of this linear model will also be helpful for interpreting the statistieal 
tables whieh are presented in this section. For this linear model, Po is the regression 
coefficient that will be associated with the independent variable (EAC SE/PM cost ratio) 
and Pi is the regression coefficient that will be associated with all other explanatory 
variables. In general, the regression coefficient (Pp represents the amount that the 
dependent variable (Y) and explanatory variable (Xp can be associated to a unit change 
in the independent variable EAC SE/PM cost ratio (Xo). The dependent variable, Y, 
represents the likelihood of a cost breach occurring. The constant (a) represents the value 
of the probability of a cost breach occurring when there are no explanatory variables (Xp 
and EAC SE/PM cost ratio (Xo) is zero (Hutcheson n.d.). Accordingly, for the first series 
of data tests, the equation for the model being tested is: 

Y = a + poXo 

Xo = EAC SE/PM cost ratio 

The hypothesis being tested is that the probability of a program sustaining a cost 
threshold breach is related to the EAC SE/PM cost ratio of the MDAP. The null 
hypothesis is that there is no relation between the probability of a cost threshold breach 
and the EAC SE/PM cost ratio and that the model has no explanatory power. 

The results of all three of the regression models without considering any other 
explanatory variables, such as the time since program initiation (Milestone B) or the type 
of contracts within the program are shown in Table 4. The OES regression and logit 
population averaged regression models indicate a positive relationship that is significant 
at the 5% level (p = 0.05) between EAC SE/PM cost ratio and the likelihood of a cost 
breach occurring. Since we established in the previous chapter that the OES regression 
does not adequately model the data and the logit fixed effects (EE) model is not 
significant, only the logit population averaged (PA) regression results will be further 
examined. 

The marginal effect of the logit population averaged model shows that, based on 
across programs observations, for every unit change in EAC SE/PM, the log odds rise in 
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the probability of a cost breach occurring is 0.80.1^ This means that for every 1% 
increase or decrease in the EAC SE/PM cost ratio, there is an average increase or 
decrease of 0.8% in the probability of a program sustaining a cost threshold breach. In 
logistic regression, the value of the marginal effect is not constant and must be 
interpreted accordingly. For example, per Table 4 when the EAC SE/PM cost ratio is 0.1 
the marginal effect on the likelihood of a cost threshold breach is 0.76. When the EAC 
SE/PM cost ratio is equal to 0.1, a 1% increase in the EAC SE/PM cost ratio will 
positively correspond to a 0.76% increase in the likelihood of a cost threshold breach 
occurring. The remaining results of the model are: 

• When the EAC SE/PM cost ratio is 0.2, a 1% change in EAC SE/PM cost 
ratio positively corresponds to a 0.90% change in the probability of a cost 
breach occurring. 

• When the EAC SE/PM cost ratio is 0.3 a 1% change in EAC SE/PM cost 
ratio positively corresponds to a 0.98% change in the probability of a cost 
breach occurring. 

• When the EAC SE/PM cost ratio is 0.4 a 1% change in EAC SE/PM cost 
ratio positively corresponds to a 1.00% change in the probability of a cost 
breach occurring. 


The logit value is the “log odds” value and is mathematieally defined to be the natural logarithm of 
the odds ratio that a eertain event will oeeur; and the odds ratio is simply the ratio of the event oeeurring 
versus the ratio that the event will not oeeur. Logit modeling is diseussed in Chapter IV, Seetion C 
“Approaeh to Analysis.” 
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EQUATION VARIABLES 

OLS 

Breach 

(Xo) 

Logit-FE 

Breach 

(Xo) 

Logit-PA 

Breach 

(Xo) 

EAC SE PM 

1.097** 

-9.238 

4.006** 

Cost Ratio (Po) 

(0.421) 

(7.825) 

(2.021) 

Constant (a) 

0.143* 


-1.473*** 


(0.0789) 


(0.454) 

Number of Observ ations 

84 

39 

84 

R-squared Statistic 

0.079 



Number of Programs 

32 

11 

32 

Robust standard errors in parentheses 
p<0.01. »• p<0.05. * p<0.1 

Average Marginal Effect in PA model is .8 


ME @EAC=0.1 0.76 
ME @ EAC=0.2 0.90 
ME @ EAC=0.3 0.98 

_ ME @EAC=0.4 1.00 _ 

Table 4. Impact of EAC SE/PM with No Explanatory Variables 

In Table 4, the number of observations represents the number of data points that 
were tested and each data point has a distinct program name and year. The “number of 
programs” is the number of programs tested in each model. Eor the OES model, all of the 
programs were tested. The r-squared statistic is a “goodness of fit” value for the model to 
the data points (no such statistic exists for logistic regression). The r-squared value is 
interpreted to mean that the independent variable, EAC SE/PM cost ratio, explains 7.9% 
of the variance in this model. It is not surprising that the r-squared statistic is so small and 
this is a testament that the single variable identified in the model (i.e., EAC SE/PM cost 
ratio) does not explain most of the variation in the dependent variable. The robust 
standard error, which is the square root of the estimate of variance for this model, is used 
instead of the standard error because it is expected to be more accurate when 
heteroskedasticity and autocorrelation exist. Heteroskedasticity describes the variance in 
the error terms of the dependent variable as the value of the explanatory variable changes 
(Barreto and Howland 2006, 508-509). Autocorrelation is the correlation of a variable 
value with itself across a time series. 

It is interesting to note that the logit fixed effects model which looks within 

programs does not show correlation between EAC SE/PM and cost breaches. This seems 
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to suggest that within a program the likelihood of a program ineurring a cost threshold 
breach cannot be associated to the EAC SE/PM cost ratio. This may be because the fixed 
effect model uses a short term within program analysis from which the change in 
likelihood for a cost breach occurring cannot be observed. It is also possible that no 
correlation is noted in the fixed effects model because more explanatory variables are 
needed to explain the relationship. Nonetheless, we can state that there is no within- 
program evidence that a change in EAC SE/PM is associated with a change in the 
likelihood of a cost breach occurring. 

The logit population averaged model provides a long term look across programs 
and yields statistically significant results for associating cost breaches to EAC SE/PM 
cost ratio. The across programs look can be interpreted as indiscriminately considering all 
Department of Defense (DoD) MDAPs in the data set, regardless of the program’s 
affiliation with a specific service or weapons type. The results suggest that, in general, 
the probability of a cost breach occurring is related to the EAC SE/PM cost ratio. That 
the logit population averaged model and the OES model both show a statistically 
significant relationship between EAC SE/PM and cost breaches is interesting; however, 
as discussed in the previous chapter, the OES regression does not account for clustered 
data. That the OES regression and logit population averaged model are both statistically 
significant may further support the claim that when considering all MDAPs, any 
randomly selected program is likely to have correlation between EAC SE/PM cost ratio 
and the probability of a cost breach. 

According to Table 4, the marginal effect of EAC SE/PM cost ratio, as defined in 
Chapter III, is a value of 0.80 for the population averaged model, for an almost one to one 
correlation between changes in EAC SE/PM cost ratio and changes in the likelihood of a 
cost breach. Eigure 9 illustrates the relationship between EAC SE/PM cost ratio and cost 
breaches for the population averaged model. The low p-value for the logit population 
averaged model (p <0.05) confirms the statistical significance of these results. In other 
words, we can assume that there is a less than 5% possibility that the results yielded in 
this study occurred by chance (Thisted 2010). Nonetheless, provided the low r-squared 
statistic (0.079) we are missing some of the explanatory variables that contribute to the 
probability a cost breach occurring. 
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Figure 9. Cost Breach versus SE/PM Logistic Regression S-Curve 

Interpretation of the regression s-curve provided (Figure 9) may lead the reader to 
correctly conclude that an increase or decrease in the FAC SE/PM cost ratio will be 
associated with a corresponding increase or decrease in the probability of a cost breach 
occurring. Again, it is necessary to recall that while there may be an association between 
EAC SE/PM cost ratio and cost breaches, there is no evidence which suggest or supports 
any claim that the EAC SE/PM is the causal factor for a cost threshold breach. 

The analysis suggests that there is a possible relation between SE/PM cost ratio 
and the likelihood of occurrence of a cost threshold breach. For example, if an MDAP 
has a SE/PM cost ratio of 0.20 we expect a 30% likelihood of incurring a cost breach. If 
that same MDAP’s SE/PM cost ratio were to increase to 0.40, the analysis suggests a 
corresponding increase in the probability of a cost breach occurring, to a 60% chance of 
incurring a cost breach. Notwithstanding, there is no evidence that the EAC SE/PM cost 
ratio should be considered a causal factor for cost breaches, merely an indicator. 

C. COMPARISON OF MDAP COST BREACHES TO SE/PM COST RATIOS 

AND PROGRAM MATURITY 

The logarithm of program maturity, considered in this report to be the length of 
time (in years) since the requisite program contracts entered Milestone B, was used to 
expand the model. As discussed in Chapter III, this definition of maturity also 
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incorporates attainment of the DoD requirements for an MDAP to aeeomplish Milestone 
B approval, namely that the design and technology utilized in the project are mature. The 
rationale behind measuring the effect of log maturity against cost breaches is that we 
expect older programs to have less uncertainty and risk than younger programs. It is 
presumed that older programs have more experienced managers and more streamlined 
and efficient processes as compared to a younger program, thus making older programs 
less likely to sustain a cost breach. Taking the natural log of maturity helps to linearize 
the relation of maturity to cost breaches in this model. In this section, we will look at the 
impaet of EAC SE/PM eost ratio on the likelihood of a eost breaeh oceurring while 
including the explanatory variable “Eog Maturity,” the logarithm of the maturity variable; 

Y = a + poXo + piXi 

Xo = EAC SE/PM Cost Ratio 

Xi = Eog Maturity 

The null hypothesis is that there is no relation of cost breaches with EAC SE/PM 
eost ratio and program maturity as defined in this report. The analysis is modeled by logit 
fixed effects and logit population averaged models and the resulting impact of the 
explanatory variable Log Maturity is displayed in Table 5. 



Logit-FE 

Logit-PA 

VARIABLES 

(XO 

(XO 

Log Maturity (Pi) 

0.429 

0.350 


(0.877) 

(0.302) 

EAC SE PM 

-8.826 

4.449** 

Cost Ratio (Po) 

(7.897) 

(1.907) 

Constant (a) 


-2.193*** 



(0.752) 

Number of Obsert'ations 

39 

84 

Number of Programs 

11 

32 

Standard errors in parentheses 


•»*p<0.01. »* 

p<0.05, * p<0.1 


Average marginal effect in PA of Log maturity is 0.7 

and EAC SE PM Cost Ratio is 0.89 

ME @ EAC=0.1 0.83 

ME @ EAC=0.2 0.99 
ME@EAC=0.3 1.08 

ME S EAC=0.4 1.08 



Table 5. Impact of EAC SE/PM with Log Maturity 

51 





While the logarithm of the length of the program does not have a statistieally 
signifieant effeet in all of the models, the average marginal effeet of Log Maturity in the 
population averaged logit model is 0.70 and for EAC SE/PM cost ratio the average 
marginal effect is 0.89. This means that a one unit (1%) change in the Eog Maturity of an 
MDAP equates to a 0.70% change in the likelihood of a cost breach occurring. Similarly, 
a one unit (1%) change in the EAC SE/PM cost ratio will result in a 0.89% change in the 
likelihood of an MDAP incurring a cost breach. The remaining results of the model are: 

• When the EAC SE/PM cost ratio is 0.1, a 1% change in EAC SE/PM cost 
ratio positively corresponds to a 0.83% change in the probability of a cost 
breach occurring. 

• When the EAC SE/PM cost ratio is 0.2, a 1% change in EAC SE/PM cost 
ratio positively corresponds to a 0.99% change in the probability of a cost 
breach occurring. 

• When the EAC SE/PM cost ratio is 0.3, a 1% change in EAC SE/PM cost 
ratio positively corresponds to a 1.08% change in the probability of a cost 
breach occurring. 

• When the EAC SE/PM cost ratio is 0.4, a 1% change in EAC SE/PM cost 
ratio positively corresponds to a 1.08% change in the probability of a cost 
breach occurring. 

The practical interpretation of these results may be that SE/PM costs become a 
larger percentage of the total program costs due to the proactive or reactive actions of the 
program manager to avoid a cost threshold breach. Thus, there is evidence to suggest that 
the SE/PM cost ratio may be an indicator of cost troubles. 

D. COMPARISON OF MDAP COST BREACHES TO SE/PM COST RATIOS, 

PROGRAM MATURITY AND CONTRACT TYPE 

We will now examine the impact of the type of contract on the likelihood of a cost 
breach occurring. Contract type was included because it is generally accepted by 
acquisitions professionals that firm fixed price contracts are frequently negotiated 
between the government and contractors for less risky programs and cost-plus contracts 
are negotiated for projects which have greater uncertainty and are perceived to be more 
risky (Government Accountability Office 2009b). Much of the rationale for associating 
contract types and projects based on the perceived riskiness of the projects is rooted in 
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the cost sharing and incentives offered by the different contract types. Nonetheless, it 
seems logical that riskier MDAPs should have cost-plus contracts and experience more 
cost breaches than their less risky counterparts. The null hypothesis of this analysis is that 
there is no relation between cost breaches and EAC SE/PM cost ratio, log maturity and 
MDAP contract type. The linear equation model which analyzes the impact of EAC 
SE/PM on the likelihood of a cost breach occurring while including the explanatory 
variables Eog Maturity and MDAP contract type is: 

Y = a + poXo + piXi + p2X2 

Xo = EAC SE/PM Cost Ratio 

Xi = Eog Maturity 

X 2 = Contract Type 

Eor this model, firm-fixed price contracts have ^2=0 and cost-plus contracts have P 2 =l. 


In a firm fixed-price contract there is not cost sharing between the government and the contractor 
and the contractor has full responsibility for the performance costs and resulting profit (or loss). In a cost- 
plus contract, a share ratio based on the contract cost and the contractor’s fee (profit) is negotiated so that 
the contractor has a pre-determined responsibility for the performance costs which will directly affect the 
fee (profit). In the case of both contract types, incentives may be offered in which the contractor’s 
responsibility for the performance costs and the profit or fee incentives offered are tailored to the 
uncertainties involved in contract performance (General Services Administration 2005). 
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Logit PA (Xi) 
VARLVBLES ‘Fixed Price" 

(P;=0) 

Logit P.A (Xi) 
“Cost Plus" 

(P:=l) 

Log Maturitv (Pi) 

-0.0967 

1.115** 


(0.586) 

(0.553) 

EAC SE PM (Pa) 

-2.121 

5.172* 

Cost Ratio 

(4.665) 

(2.846) 

Constant (a) 

-0.906 

-3.255*** 


(1.346) 

(1.214) 

Number of Observations 

3"^ 

41 

Number of Programs 

15 

16 


Robust standard errors in parentheses 
*** p<0-01, ** p<0.05. * p<0.1 


Including \IE for the t>pe 1 model only 
Average marginal effect in PA of Log maturity is .21, and EACSEP is 1.00 

ME 5 EAC=0.1 0.94 
ME S EAC=0.2 1.08 
ME I EAC=0.3 1.12 

_ ME I EAC=0.4 1.07 _ 


Table 6. Logistic Regression with Log Maturity and Contract Type 


Noting the results summarized in Table 6 and considering the independent and 
explanatory variables in this model, the statistically significant results are observed in 
programs with cost-plus contracts. This observation is statistically significant at the 10% 
level. Since only cost-plus programs yield statistically significant results, the marginal 
effects of the independent and explanatory variables have only been measured for 
programs with cost-plus contracts. That the model does not produce statistically 
significant results for programs with fixed-price contracts implies the existence of 
additional explanatory variables which are not captured within the scope of this report. 
For the cost-plus contract programs, according to the population averaged model, the 
average marginal effect of Log Maturity is 0.21 and for EAC SE/PM cost ratio is 1.00. 
This means that for a one unit (1%) change in the Eog Maturity the likelihood of a cost 
breach occurring will change by 0.21%. Moreover, for a one unit (1%) change in the 
EAC SE/PM cost ratio the likelihood of a cost breach occurring will change by 1.00%. 
Similar to the results in Section C of this chapter, as the EAC SE/PM cost ratio changes 
from 0.1 to 0.4 there are corresponding changes in the marginal effect on the probability 
of a cost breach occurring in a program; 
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• When the EAC SE/PM eost ratio is 0.1, a 1% change in the EAC SE/PM 
cost ratio positively corresponds to a 0.94% change in the probability of a 
cost breach occurring. 

• When the EAC SE/PM cost ratio is 0.2, a 1% change in the EAC SE/PM 
cost ratio positively corresponds to a 1.08% change in the probability of a 
cost breach occurring. 

• When the EAC SE/PM cost ratio is 0.3, a 1% change in the EAC SE/PM 
cost ratio positively corresponds to a 1.12% change in the probability of a 
cost breach occurring. 

• When the EAC SE/PM cost ratio is 0.4, a 1% change in the EAC SE/PM 
cost ratio positively corresponds to a 1.07% change in the probability of a 
cost breach occurring. 

E. SUMMARY OF RESULTS 

In this chapter we looked at the SE/PM ratios and cost breaches for 32 MDAPs 
during the period of 1998 to 2011. We noted that the observed SE/PM cost ratios were 
approximately normally distributed with an average of 0.16 and a range of 0.05 to 0.5. 
Our graphical analysis indicated that there appears to be a relationship between SE/PM 
cost ratios and cost breaches as shown in Eigure 8. We used multiple models and three 
types of regression to further examine this relationship: ordinary least squares (OES) 
regression, fixed effects (EE) logit and population averaged (PA) logit, as described in 
Chapter IV. A linear model was used to describe the relationship: 

Y = a + poXo + piXi + p2X2 

Xo = EAC SE/PM Cost Ratio 

Xi = Eog Maturity 

X 2 = Contract Type 

Based on the results of the analysis, using the population averaged logit model to 
look across programs seems most useful. This is probably due to the population averaged 
model making longer term observations, whereas the fixed effects model is making short 
term observations. Of the three variables tested using the various logistic regression 
techniques, the model which included the explanatory variables Eog Maturity and cost- 
plus contract types had the largest values for the average marginal effects of EAC SE/PM 
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cost ratio; the average marginal effeet of Log Maturity is 0.21 and for EAC SE/PM eost 
ratio is 1.00. Eurthermore, in this model as the EAC SE/PM eost ratio ehanges from 0.1 
to 0.4 there are eorresponding ehanges in the marginal effeet on the probability of a eost 
breaeh oeeurring in a program. 

The analysis results seem to support the hypothesis that high transaetion eosts are 
an ex-ante indieator related to MDAP eost breaehes. The next ehapter will diseuss 
eonelusions that ean be drawn from these results, as well as researeh areas whieh were 
not eaptured in this report that may eontribute to improving DoD eost estimation. 
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VI. SUMMARY AND FUTURE RESEARCH 


This chapter of the report provides an overview of the salient observations of this 
research. This includes inferences that can be made based on the key takeaways from the 
logit population averaged regression, which turned out to be the best method of approaeh 
that was eonsidered for modeling the data. As this research only addresses the role that 
transaction costs play in Department of Defense (DoD) cost estimates, this ehapter also 
presents areas for further researeh that may help system engineers and cost analysts to 
improve program eost estimation and possibly use breaehes to create a predictive model 
in the future. 

A. CONCLUSIONS 

This analysis is based on the assumption that there is a real cost for business 
negotiations and managing the program, commonly known as transaction costs. We seek 
to determine (1) if there is a correlation between these transaction costs and acquisition 
cost threshold breaches; and (2) if cost threshold breaches are related to the amount, or 
proportion of total costs, that are spent on program transaction costs. The data set 
included major defense aequisition programs (MDAPs) with 10% (or greater) cost 
threshold breaehes. Using logit population averaged regression analysis yielded 
statistieally significant results for the programs with cost-plus contracts. These contraets 
exhibited relationships between (1) the estimate at eompletion (EAC) system 
engineering/program management (SE/PM) eost ratio and the likelihood of a cost breach 
occurring; and (2) the log maturity and the likelihood of a cost breach occurring. 

The results show that as the EAC SE/PM cost ratio rises between 0.1 and 0.3 
there is a statistieally signifieant eorresponding increase in the probability of a cost 
threshold breach occurring. This provides answers to both of these researeh questions. 
Eirst, the statistically significant correspondence between the variables infers that 
eorrelation exists. Second, the relation between cost threshold breaches and the amount of 
transaction costs is confirmed because as the relative amount of transaction costs for an 
MDAP increases, the likelihood of the oceurrenee of a eost threshold breaeh also 
increases. 
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The report also sought to determine whether underestimated transaetion eosts had 
eontributed to MDAP eost threshold breaehes. While the research established correlation 
between the independent and dependent variables there was no testing for causality. The 
existence of a correlation between transaction costs and MDAP cost threshold breaches 
neither suggests that increased transaction costs caused the cost threshold breach, nor that 
a cost threshold breach generated increased transaction costs. Without establishing 
causality, it cannot conclusively be stated that underestimated transaction costs have 
contributed to cost growth for MDAPs. 

B. IMPLICATIONS TO SYSTEMS ENGINEERING 

It is incumbent upon system engineers (SEs) to understand the cost estimation 
process for any project or program in order to provide support to the program manager. 
The SE process is the overarching framework designed to be the “integrating mechanism 
for... limitations imposed by technology, budget, and schedule” (Defense Acquisition 
University 2011b, B-264). Eor the SE effort to be effective, understanding the impact of 
transaction costs on the acquisition cost of systems is important. Our research informs the 
SE process by suggesting an association between transaction costs and cost threshold 
breaches for major defense acquisition programs (MDAPs) with cost-plus contracts. 

C. AREAS FOR FURTHER RESEARCH 

The logit population averaged model proved to be useful in this research for 
multiple reasons. Two of the more salient reasons are that (1) logistic regression models 
are designed for binary-outcome dependent variables; and (2) the data is in panel form, as 
discussed in Chapter IV. This research strongly suggests that of the three models 
considered, the logit population averaged model would be useful for future research— 
particularly for estimating the marginal effect of the independent variable(s) on the 
dependent variable. That is, the logit population averaged model would be useful for 
testing the impact of EAC SE/PM cost ratio with other explanatory variables against the 
probability of a cost threshold breach occurring. 

It would be interesting to run logit regressions with more variables in order to 

increase the explanatory power of the model and determine other factors which may 
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contribute to cost threshold breaches. That the “maturity” of the program, as defined in 
Chapter IV (Section B) of this report, is related to the occurrence of a cost threshold 
breach is interesting and may be worthy of further research. In particular, the correlation 
of log maturity paired with other variables to cost breaches has not yet been explored. 
Whether by using actual data or proxies, future research of the following for correlation 
to cost threshold breaches may prove interesting: (I) program complexity; (2) “to date” 
SE/PM cost ratio; and (3) the degree to which programs are ahead of or behind on 
schedule. 

Tracking MDAP SE/PM cost ratio trends over time to better understand the 
implications of pre-breach SE/PM cost ratios and post-breach SE/PM cost ratios may also 
prove useful to program managers. Perhaps the pre-breach SE/PM cost ratio when compared 
against program acquisition milestones through time is indicative of future breaches. Is it 
possible that the SE/PM cost ratio is a resource tail for mitigating actions that a program 
manager may take to avoid or lessen the severity of a cost threshold breach? 

Ultimately, considering the association of these independent variables with the 
dependent variable will be integral to developing a multivariate model that has predictive 
power for MDAP cost threshold breaches. Such a predictive model would be an 
invaluable tool for MDAP stakeholders and particularly systems engineers and program 
managers. It is likely that any future research along the lines essayed in this study 
involves revisiting the basic model—^which is depicted in Eigure 2. While it proved 
useful, it (like all models) is a simplification of the real world. And, it is likely this 
particular effort has wrung out about all that model can provide by way of insights and 
explanatory power. Hence, any significant advances in the understanding of transaction 
costs in defense costs, and their estimation, probably entails the formulation of a richer 
(and likely more complex) model. That said, we feel that Eigure 2 does provide a useful 
starting point for devising a better model. 
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APPENDIX 


A. DEFINITION OF TERMS AND VARIABLES 

All definitions from DAU Glossary, 2011 found at (https://dap.dau.mil/glossary/ 
Pages/Default, aspx) unless otherwise indieated. 


Acquisition Cost Estimate 


Component Cost Estimate 
(CCE). 


Equal to the sum of the development cost for prime 
mission equipment and support items; the 
procurement cost for prime mission equipment, 
support items, and initial spares; and the system 
specific facilities cost. 

The generic term —DoD Component Cost Estimatell 
is used to provide considerable latitude to each 
military service or defense agency as to the actual 
responsibility for this cost estimate. In some cases, a 
military service assigns the responsibility to the 
Program Office (PO), which then provides a 
Program Office Eife Cycle Cost Estimate (PECCE). 
In other cases, the DoD Component may adopt a 
more corporate approach in which an initial PO cost 
estimate is subject to considerable review and 
possible adjustment as determined by the Service 
Cost Center or defense agency equivalent (Defense 
Acquisition Guidebook 2011a). 


Cost Estimate A judgment or opinion regarding the cost of an 

object, commodity, or service. A result or product of 
an estimating procedure that specifies the expected 
dollar cost required to perform a stipulated task or to 
acquire an item. A cost estimate may constitute a 
single value or a range of values. 

Estimate at Completion Actual direct costs, plus indirect costs or costs 

(EAC) allocable to the contract, plus the estimate of costs 

(direct and indirect) for authorized work remaining. 


Independent Cost Estimate A Eife Cycle Cost Estimate (ECCE) for Acquisition 

(ICE). Category (ACAT) I programs prepared by an office 

or other entity not under the supervision, direction, 
or control of the military department, defense 
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agency, or other Component of DoD that is directly 
responsible for B-116 development or acquisition of 
the program, or if the decision authority has been 
delegated to a Component, prepared by an office or 
other entity that is not directly responsible for 
carrying on the development or acquisition of the 
program. 


Independent Government 
Cost Estimate (IGCE). 


Eife Cycle Cost 


Nunn-McCurdy Breach 


Operation and Maintenance 
(O&M) 


An estimate of the cost for goods and/or estimate of 
services to be procured by contract. Such estimates 
are prepared by government personnel, i.e., 
independent of contractors. 

Eor a defense acquisition program, LCC consists of 
Research and Development (R&D) costs, investment 
costs, operating and support costs, and disposal costs 
over the entire life cycle. These costs include not 
only the direct costs of the acquisition program, but 
also indirect costs that would be logically attributed 
to the program. In this way, all costs that are 
logically attributed to the program are included, 
regardless of funding source or management control 
(Defense Acquisition Guidebook 2011a). 

Refers to Title 10 U.S.C. § 2433, Unit Cost Reports 
(UCRs). This amendment to Title 10 was introduced 
by Senator Sam Nunn and Representative Dave 
McCurdy in the National Defense Authorization Act 
(NDAA) for Eiscal Year (EY) 1982. Requires that 
Acquisition Category I (ACAT I) Program Managers 
(PMs) maintain current estimates of Program 
Acquisition Unit B-174 Cost (PAUC) and Average 
Procurement Unit Cost (APUC). If the PAUC or 
APUC increases by 25 percent or more over the 
current Acquisition Program Baseline (APB) 
objective, or 50 percent or more over the original 
APB objective, the program must be terminated 
unless the Secretary of Defense (SECDEE) certifies 
to Congress that the program is essential to national 
security (DoDI 5000.02 and Defense Acquisition 
Guidebook, Chapter 10). 

O&M appropriations fund expenses such as civilian 
salaries, travel, minor construction projects. 
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Procurement 


Program Offiee Estimate 
(POE). 

Research, Development, 
Test, and Evaluation 
(RDT&E) 


SE/PM Cost 


SE/PM Ratio 


operating military forees, training and education, 
depot maintenanee, stock funds, and base operations 
support. 


Procurement appropriations fund those aequisition 
programs that have been approved for production (to 
include Eow-Rate Initial Production (ERIP) of 
acquisition objective quantities), and all costs 
integral and necessary to deliver a useful end item 
intended for operational use or inventory upon 
delivery. 

A Component Cost Estimate (CCE) of Eife Cyele 
Costs (ECCs) eondueted by an aequisition Program 
Office (PO). 

RDT&E appropriations fund the efforts performed 
by contractors and government activities required for 
the Research and Development (R&D) of 
equipment, material, computer applieation software, 
and their Test and Evaluation (T&E) including 
Initial Operational Test and Evaluation (lOT&E) and 
Eive Eire Test and Evaluation (EET&E). RDT&E 
also funds the operation of dedicated R&D 
installation activities for the conduct of R&D 
programs. 

This cost is the sum of reeurring and non-reeurring 
costs for the SE/PM WBS line item. This cost is not 
inclusive of the contractor’s or subcontractor’s 
profit/loss or fees. 

This ratio serves as a proxy for transaction costs and 
is defined to be the program’s SE/PM eosts divided 
by total cost: 


SE/PM Ratio = 


"To Date" SE/PM Costs 
Total Cost EAC 
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B. USING THE DEFENSE ACQUISITION MANAGEMENT INFORMATION 

RETRIEVAL SYSTEM 

Establishing an account with the Defense Aequisition Management Information 
Retrieval (DAMIR) system was a fairly simple proeess. Individual institutions are 
assigned DAMIR points of contact (POCs) who are responsible for requesting a DAMIR 
aceount on behalf of the user. The user is responsible for providing the requisite personal 
information and justification for use to the institution’s DAMIR POC. The DAMIR help 
desk (phone number found at http://www.aoq.osd.mil/damir/index.html) was helpful for 
establishing the DAMIR account. 

Generally, the organized and established criteria for SAR submissions faoilitated 
the effort for finding MDAPs and their assooiated SARs, but it is important to note that 
political events can also affect SAR submission. For instance, in years which the 
president’s budget was signed late (e.g., 2000 and 2008) the only SARs found are for 
MDAPs whioh met the mandate to submit an interim quarterly SAR, such as a Nunn- 
McCurdy breach or schedule delay of at least 6 months. 
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C. USING THE DEFENSE COST AND RESOURCE CENTER 

It requires approximately one week to establish a Defense Cost and Resouree 
Center (DCARC) aeeount and aeeess the eost and software data reporting (CSDR) 
database. Using the online registration instruetions found at the DCARC Knowledge 
Portal (http://doaro.cape.osd.mil/) made the process simple. The access levels that are 
required to view the CSDR information used in this report are “DACIMS35_Analyst” 
and “EVM Analyst.” Once the user’s CAC card is registered with DCARC, access will 
be granted to the database. Access to DCARC also includes access to the Defense 
Automated Cost Information Management Systems (DACIMS) 3.5 Library. The CSDR 
library contains folders of active and inactive MDAP contract data sorted by weapon 
system types. More specifically, contract information within each of these folders 
includes: the DD Form 1921 (Cost Summary Data Report), the DD Form 1921-1 
(Functional Cost-Hour Report) and the contract work breakdown structure (CWBS) 
Index and Dictionary among other documents. 

1. Sample Cost Data Summary Report (DD Form 1921) 

The cost data summary report (Figure 10) comprises many categories of program 
metadata and reported data. A list of these categories and detailed information about the 
DD Form 1921 can be found in DI-FNCL-81565C. 


67 




DD FORM 1921, APR 2007 PREVIOUS EDITION IS OBSOLETE SECURITY CLASSIFICATION Aowpwiwon-f o 


Figure 10. Cost Data Summary Report (Blank) 
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2. Steps for Estimate at Completion SE/PM Cost Ratio Calculation 

• Determine the SE/PM eost. Identify the line item for “Systems 
Engineering/Program Management” in the eolumn for “WBS Reporting 
Elements” (Column B). Record the “Total” estimate at completion cost for 
Systems Engineering/Program Management (Column J). 

• Determine the EAC total cost. Identify the “Total Cost (Eess Profit or 
Pee)” in the column for “WBS Reporting Elements” (Column B). Record 
the “Total” estimate at completion cost for EAC total cost (Column J). 

• Determine the SE/PM cost ratio. Divide the SE/PM cost (Step 1) by the 
EAC total cost (Step 2). Record the SE/PM cost ratio value. 
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D. PROGRAM LIST AS REPORTED IN SELECTED ACQUISTION 
REPORT (SAR) 

Active Electronically Scanned Array (AESA) Radar 
AIM-9X/Short Range Air-to-Air Missile 

AIM-120 Advaneed Medium Range Air-to-Air Missile (AMRAAM) 

Airborne and Maritime/Eixed Station Joint Tactical Radio System (AME JTRS) 
AN/WQR-3, Advaneed Deployable System (ADS) 

Apache Block IIIA Remanufacture (AB3A REMANUEACTURE) 

AV-8B/Attack, V/STOE, Close Air Support (Harrier 11+ Remanufacture) 

B-2 Radar Modernization Program 

Cobra Judy Replaeement (Cobra Judy Replaeement) 

EA-18G Growler (EA-18G) 

Expeditionary Eighting Vehiele (EEV) 

E-3 AWACS Radar System Improvement Program (RSIP) 

E-2C Reproduction 

Eamily of Advanced Beyond Eine-of-Sight Terminals (EAB-T) 

Eamily of Medium Tactical Vehicles (EMTV) 

Guided Multiple Eaunch Roeket System/DPICM/Unitary/Alternative Warhead 
(GMLRS/GMLRS AW) 

Joint Common Missile (JCM) 

Joint Taetieal Radio System Ground Mobile Radio (formerly Cluster 1) 

(JTRS GMR) 

Eongbow Hellfire - subsystem of the AH-64 Apache Weapon System 
EHA Replacement Amphibious Assault Ship 

MQ-4C Unmanned Aircraft System Broad Area Maritime Surveillanee (MQ-4C 
UAS BAMS) 

Multi-Platform Radar Technology Insertion Program (MP-RTIP) 

National Polar-orbiting Operational Environmental Satellite System (NPOESS) 
Presidential Helicopter Replaeement (VH-71) Program 
P-8A Poseidon 

Sense and Destroy Armor (SADARM) 

Small Diameter Bomb Increment II (SDB II) 

Spaee Based Infrared System (SBIRS) High Program 
Standard Missile (SM) - 2 Bloek IV 
Stryker Eamily of Vehicles (STRYKER) 

UH-72A Eight Utility Helieopter (EUH) 

Warfighter Information Network - Taetieal (WIN-T) 
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